{'fraction_non_alphanumeric': 0.03161179591199875, 'fraction_numerical': 0.010672491808394446, 'mean_word_length': 4.986549598356062, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos

We propose an end-to-end learning framework for segmenting generic objects in videos. Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos. We formulate this task as a structured prediction problem and design a two-stream fully convolutional neural network which fuses together motion and appearance in a unified framework. Since large-scale video datasets with pixel level segmentations are problematic, we show how to bootstrap weakly annotated videos together with existing image recognition datasets for training. Through experiments on three challenging video segmentation benchmarks, our method substantially improves the state-of-the-art for segmenting generic (unseen) objects.


I. INTRODUCTION
In video object segmentation, the task is to separate out foreground objects from the background across all frames. This entails computing dense pixel level masks for foreground objects, regardless of the object's category-i.e., learned objectspecific models must not be assumed. A resulting foreground object segment is a spatio-temporal tube delineating object boundaries in both space and time. This fundamental problem has a variety of applications, including high level vision tasks such as activity and object recognition, as well as graphics areas such as post production video editing and rotoscoping.
While appearance alone drives segmentation in images, videos provide a rich and complementary source of information in form of object motion. It is natural to expect that both appearance and motion should play a key role in successfully segmenting objects in videos. However, existing methods fall short of bringing these complementary sources of information together in a unified manner.
In particular, today motion is employed for video segmentation in two main ways. On the one hand, the propagation or interactive techniques strongly rely on appearance information stemming from human-drawn outlines on frames in the video. Here motion is primarily used to either propagate information or enforce temporal consistency in the resulting segmentation [13], [14], [15], [20]. On the other hand, fully automatic methods strongly rely on motion to seed the segmentation process by locating possible moving objects. Once a moving object is detected, appearance is primarily used to track it across frames [4], [6], [8], [9]. Such methods can fail if the object(s) are static or when there is significant camera motion. In either paradigm, results suffer because the two essential cues are treated only in a sequential or disconnected way.
We propose an end-to-end trainable model that draws on the respective strengths of generic (non-category-specific) object appearance and motion in a unified framework. Specifically, we develop a novel two-stream fully convolutional deep segmentation network where individual streams encode generic appearance and motion cues derived from a video frame and its corresponding optical flow. These individual cues are fused in the network to produce a final object versus background pixellevel binary segmentation for each video frame. The proposed approach segments both static and moving objects without any human involvement.
Declaring that motion should assist in video segmentation is non-controversial, and indeed we are certainly not the first to inject motion into video segmentation, as noted above. However, thus far the sum is not much greater than its parts. We contend that this is because the signal from motion is adequately complex such that rich learned models are necessary to exploit it. For example, a single object may display multiple motions simultaneously, background and camera motion can intermingle, and even small-magnitude motions should be informative (see Fig. 1).
To learn the rich signals, sufficient training data is needed. However, no large-scale video datasets of pixel-level segmentation masks exist. A second contribution of our work is to address this practical issue. We propose a solution that leverages readily available image segmentation annotations together with weakly annotated video data to train our model.
Our results show the reward of learning from both signals in a unified framework: a true synergy, with substantially stronger results than what we can obtain from either one alone-even if they are treated with an equally sophisticated deep network. We significantly advance state of the art for fully automatic video object segmentation on multiple challenging datasets. In some cases, the proposed method even outperforms existing methods that require manual intervention.
In summary our key contributions are: • the first end-to-end learning framework for producing pixel level foreground object segmentation in videos. • state-of-the-art on multiple datasets, improving over many reported results in the literature and strongly outperforming simpler applications of optical flow, and • a means to train a deep pixel-level video segmentation model with access to only weakly labeled videos and strongly labeled images, with no explicit assumptions about the categories present in either.

II. RELATED WORK
Automatic methods: Fully automatic or unsupervised video segmentation methods assume no human input on the video. They can grouped into two broad categories. First we have the supervoxel methods [1], [2], [3] which oversegment the video volume into space-time blobs with cohesive appearance and motion. Their goal is to generate mid-level video regions useful for downstream processing, whereas ours is to produce space-time tubes which accurately delineate object boundaries. Second we have the fully automatic methods that generate thousands of "object-like" space-time segments [21], [22], [23], [24], [25].
While useful in accelerating object detection, it is not straightforward to automatically select the most accurate one when a single hypothesis is desired. Methods that do produce a single hypothesis [4], [5], [6], [8], [9], [26], [27], [28] strongly rely on motion to identify the foreground objects, either by seeding appearance models with moving regions or directly reasoning about occlusion boundaries using optical flow. This limits their capability to segment static objects in video. In comparison, our method is fully automatic, produces a single hypothesis, and can segment both static and moving objects.
Human-guided methods: Semi-supervised label propagation methods accept human input on a subset of frames, then propagate it to the remaining frames [10], [11], [29], [12], [13], [14], [15], [20], [30]. In a similar vein, interactive video segmentation methods leverage a human in the loop to provide guidance or correct errors, e.g., [16], [18], [19], [31]. Since the human pinpoints the object of interest, these methods typically focus more on learning object appearance from the manual annotations. Motion is primarily used to propagate information or enforce temporal smoothness. In the proposed method, both motion and appearance play an equally important role, and we show their synergistic combination results in a much better segmentation quality. Moreover, our method is fully automatic and uses no human involvement.
Category-specific semantic segmentation: State-of-theart semantic segmentation techniques for images rely on fully convolutional deep learning architectures that are end-to-end trainable [32], [33], [34], [35]. These deep learning based methods for segmenting images have seen rapid advances in recent years. Unfortunately, video segmentation has not seen such rapid progress. We hypothesize that the lack of largescale human segmented video segmentation benchmarks is a key bottleneck. Recent video benchmarks like Cityscapes [36] are valuable, but 1) it addresses category-specific segmentation, and 2) thus far methods competing on it process each frame independently, treating it like multiple image segmentation tasks. 1 In contrast, we aim to segment generic objects in video, whether or not they appear in training data. Furthermore, our idea to leverage weakly labeled video for training opens a path towards training deep segmentation models that fuse spatial and temporal cues.
Deep learning with motion: End to end deep learning for combining motion and appearance in videos has proven to be useful in several other computer vision tasks such as video classification [38], [39], action recognition [40], [41], object tracking [42], [43], [44] and even computation of optical flow [45]. While we take inspiration from these works, we are the first to present a unifying deep framework for segmenting objects in videos.

III. APPROACH
Our goal is to segment generic objects in video, independent of the object categories they belong to, and without any manual intervention. We pose the problem as a dense labeling task: given a sequence of video frames [I 1 , I 2 , ..., I N ], we want to infer either "object" or "background" for each pixel in each frame, to output a sequence of binary maps [S 1 , S 2 , ..., S N ].
We propose a solution based on a convolutional neural network.
First we segment generic objects based on appearance only from individual frames (Sec. III-A). Then we use the appearance model to generate initial pixel-level annotations in training videos, and bootstrap strong annotations to train a model from motion (Sec. III-B). Finally, we fuse the two streams to perform video segmentation (Sec. III-C).

A. Appearance Stream
Building on the "pixel objectness" method of [46], we train a deep fully convolutional network to learn a model of generic foreground appearance. The main idea is to pre-train for object classification, then re-purpose the network to produce binary object segmentations by fine-tuning with relatively few pixel-labeled foreground masks. The authors use the VGG  architecture [47] and transform its fully connected layers into convolutional layers. As shown in [46], the resulting network possesses a strong notion of objectness, making it possible to identify foreground regions of more than 3,000 object categories despite seeing ground truth masks for only 20 during training. We take this basic idea and upgrade its implementation for our work. In particular, we adapt the image classification model ResNet-101 [37], [48] by replacing the last two groups of convolution layers with atrous convolution layers (also known as dilated convolution) to increase feature resolution. This results in only an 8× reduction in the output resolution instead of a 32× reduction in the output resolution in the original ResNet model. In order to improve the model's ability to handle both large and small objects, we replace the classification layer of ResNet-101 with four parallel atrous convolutional layers with different sampling rates to explicitly account for object scale. Then we fuse the prediction from all four parallel atrous convolutional layers by summing all the outputs. The loss is the sum of cross-entropy terms over each pixel position in the output layer, where ground truth masks consist of only two labels-object foreground or background. We train the model using the Caffe implementation of [48]. The network takes a video frame of arbitrary size and produces an objectness map of the same size. See Fig 2. While our method leverages pixel objectness [46] for the appearance stream, from this point forward in the paper our work cleanly departs from that contribution. Whereas [46] tackles images, we tackle video. The novelty of [46] is to learn an objectness model from few objects that generalizes to many others, with applications to image retrieval and retargeting. The novelty of this paper is the first deep learning framework for video object segmentation, and a means to train deep video segmentation models with weakly labeled video and strongly labeled images.

B. Motion Stream
Our complete video segmentation network architecture consists of a two-stream network in which parallel streams for appearance and motion process the RGB and optical flow images, respectively, then join in a fusion layer (see Fig. 2).
The direct parallel to the appearance stream discussed above would entail training the motion stream to map optical flow maps to video frame foreground maps. However, an important practical catch to that solution is training data availability. While ground truth foreground image segmentations are at least modestly available, datasets for video object segmentation masks are small-scale in deep learning terms, and primarily support evaluation. For example, Segtrack-v2 [7], one of the most commonly used benchmark datasets for video segmentation, contains only 14 videos with 1066 labeled frames. DAVIS [49] contains only 50 sequences with 3455 labeled frames. None contain enough labeled frames to train a deep neural network. Semantic video segmentation datasets like CamVid [50] or Cityscapes [51] are somewhat larger, yet limited in object diversity due to a focus on street scenes and vehicles.
A good training source for our task would have ample frames with human-drawn segmentations on a wide variety of foreground objects, and would show a good mix of static and moving objects. No such large-scale dataset exists and creating one is non-trivial.
We propose a solution that leverages readily available image segmentation annotations together with weakly annotated video data to train our model. In brief, we temporarily decouple the two streams of our model, and allow the appearance stream to hypothesize likely foreground regions in frames of a large video dataset annotated only by bounding boxes. Since the appearance alone need not produce perfect segmentations in video, we devise a series of filtering stages by which the system zeros in on high quality estimates of the true foreground. These instances bootstrap pre-training of the optical flow stream, then the two streams are joined to learn the best combination from minimal human labeled training videos.
More specifically, given a video dataset with bounding boxes labeled for each object, 2 we ignore the category labels and map the boxes alone to each frame. Then, we apply the appearance stream, thus far trained only from images labeled by their foreground masks, to compute a binary segmentation for each frame.
Next we deconflict the box and segmentation in each training frame. First, we refine the binary segmentation by setting We first apply the appearance model to obtain initial segmentations (second row, with object segment in green) and then prune by setting pixels outside bounding boxes as background (third row). Then we apply the bounding box test (fourth row, yellow bounding box is ground truth and blue bounding box is the smallest bounding box enclosing the foreground segment) and optical flow test (fifth row) to determine whether we add the segmentation to the motion stream's training set or discard it. Best viewed in color.
all the pixels outside the bounding box(es) as background. Second, for each bounding box, we check if the the smallest rectangle that encloses all the foreground pixels overlaps with the bounding box by at least 75%. Otherwise we discard the segmentation. Third, we discard regions where the box contains more than 95% pixels labeled as foreground, based on the prior that good segmentations are rarely a rectangle, and thus probably the true foreground spills out beyond the box. Finally, we eliminate segments where object and background lack distinct optical flow, so our motion model can learn from the desired cues. Specifically, we compute the frame's optical flow using [52] and convert it to an RGB flow image [53]. If the 2-norm between a) the average value within the bounding box and b) the average value in a box whose height and width are twice the original size exceeds 30, the frame and filtered segmentation are added to the training set. 3 See Fig 3 for visual illustration of these steps.
To recap, bootstrapping from the preliminary appearance model, followed by bounding box pruning, bounding box tests, and the optical flow test, we can generate accurate per-pixel foreground masks for thousands of diverse moving objects-for which no such datasets exist to date. Note that by eliminating training samples with these filters, we aim to reduce label noise for training. However, at test time our system will be evaluated on standard benchmarks for which each frame is manually annotated (see Sec. IV).
With this data, we now turn to training the motion stream. Analogous to our strong generic appearance model, we also want to train a strong generic motion model that can segment foreground objects purely based on motion. We use exactly the same network architecture as the appearance model (see Fig. 2). Our motion model takes only optical flow as the input and is trained with automatically generated pixel level ground truth segmentations. In particular, we convert the raw optical flow to a 3-channel (RGB) color-coded optical flow image [53]. We use this color-coded optical flow image as the input to the motion network. We again initialize our network with pre-trained weights from ImageNet classification [54]. Representing optical flow using RGB flow images allows us to leverage the strong pre-trained initializations as well as maintain symmetry in the appearance and motion arms of the network.
An alternative solution might forgo handing the system optical flow, and instead input two raw consecutive RGB frames. However, doing so would likely demand more training instances in order to discover the necessary cues. Another alternative would directly train the joint model that combines both motion and appearance, whereas we first "pre-train" each stream to make it discover convolutional features that rely on appearance or motion alone, followed by a fusion layer (below). Our design choices are rooted in avoiding bias in training our model. Since the (pseudo) ground truth comes from the initial appearance network, either supplying two consecutive RGB frames or training jointly from the onset is liable to bias the network to exploit appearance at the expense of motion. By feeding the motion model with only optical flow, we ensure our motion stream learns to segment objects from motion.

C. Fusion Model
The final processing in our pipeline joins the outputs of the appearance and motion streams, and aims to leverage a whole that is greater than the sum of its parts. We now describe how to train the joint model using both streams.
An object segmentation prediction is reliable if 1) either our appearance model or our motion model predicts the object segmentation with very strong confidence 2) both our appearance model and our motion model predict the segmentation. This motivates the network structure of our joint model. We implement the idea by creating three independent parallel branches: 1) We apply a 1×1 convolution layer followed by a RELU to the output of the appearance model 2) We apply a 1×1 convolution layer followed by a RELU to the output of the motion model 3) We replicate the structure of first and second branches and apply element-wise multiplication on their outputs. The element-wise multiplication ensures the third branch outputs confident predictions of object segmentation if and only if both appearance model and motion model have strong predictions. We finally apply a layer that takes the element-wise maximum to obtain the final prediction. See Fig.  2.
As discussed above, we do not fuse the two streams in an early stage of the networks because we want them both to have strong independent predictions. Another advantage of our approach is we only introduce six additional parameters in each 1×1 convolution layer, for a total of 24 trainable parameters. We can then train the fusion model with very limited annotated video data, without overfitting.

Datasets and metrics:
We evaluate our method on three challenging video object segmentation datasets: DAVIS [49], YouTube-Objects [55], [56], [57] and Segtrack-v2 [7]. To measure accuracy we use the standard Jaccard score, which computes the intersection over union overlap (IoU) between the predicted and ground truth object segmentations. The three datasets are: • DAVIS [49]: the latest and most challenging video object segmentation benchmark consisting of 50 high quality video sequences of diverse object categories with 3, 455 densely annotated, pixel-accurate frames. The videos are unconstrained in nature and contain challenges such as occlusions, motion blur, and appearance changes. While the videos contain both static and moving objects, only the prominent moving objects were annotated in the ground-truth. • YouTube-Objects [55], [56], [57]: consists of challenging Web videos from 10 object categories and is commonly used for evaluating video object segmentation. We use the subset defined in [57] and the ground truth provided by [56] for evaluation. • SegTrack-v2 [7]: one of the most common benchmarks for video object segmentation consisting of 14 videos with a total of 1, 066 frames with pixel-level annotations. For videos with multiple objects with individual groundtruth segmentations, we treat them as a single foreground for evaluation. Baselines: We compare with several state-of-the-art methods for each dataset as reported in the literature. Here we group them together based on whether they can operate in a fully automatic fashion (unsupervised) or require a human in the loop (semi-supervised) to do the segmentation: • Unsupervised methods: Unsupervised video segmentation methods do not require any human involvement to segment new videos. Depending on the dataset, we compare with the following top-performing state of the art methods: FST [8], KEY [4], NLC [9] and COSEG [26]. All use some form of unsupervised motion or objectness cues to identify foreground objects followed by postprocessing to obtain spatio-temporal object segmentations. • Semi-supervised methods: Semi-supervised methods bring a human in the loop. They typically have some knowledge about the object of interest which is exploited to obtain the segmentation. We compare with the following state-of-the-art methods: HVS [1], HBT [58], FCP [20], IVID [19], HOP [56], and BVS [30]. The methods require different amounts of human annotation time to operate, e.g. HOP, BVS, and FCP make use of manual complete object segmentation in the first frame to seed the method; HBT requests a bounding box around the object of interest in the first frame; HVS, IVID require a human to constantly guide the algorithm whenever it starts to fail.
Note that our method requires human annotated data only during training. At test time it operates in a fully automatic fashion. Thus, given a new video, we require equal effort as the unsupervised methods, and less effort than the semi-supervised methods.
Apart from these comparisons, we also examine some natural baselines and variants of our method: • Flow-thresholding (Flow-Th): To examine the effectiveness of motion alone in segmenting objects, we adaptively threshold the optical flow in each frame using the flow magnitude. Specifically, we compute the mean and standard deviation from the L2 norm of optical flow magnitude and use "mean+unit std." as the adaptive threshold. Implementation details: To train the appearance stream we rely on the PASCAL VOC 2012 segmentation dataset [60] and use a total of 10,582 training images with binary object vs. background masks (see [46] for more details). As weak bounding box video annotations, we use the ImageNet-Video dataset [61]. This dataset comes with a total of 3,862 training videos from 30 object categories with 866,870 labeled object bounding boxes from over a million frames. Post refinement using our ground truth generation procedure (see Sec. III-B), we are left with 84,929 frames with good pixel segmentations which are then used to train our motion model. For training the joint model we use a leave-one-dataset out strategy. No postprocessing is applied on the segmentations obtained from our networks. Quality of training data: To ascertain that the quality of training data we automatically generate for training our motion stream is good, we first compare it with a small amount of human annotated ground truth. We randomly select 100 frames that passed both the bounding box and optical flow tests, and collect human-drawn segmentations on Amazon MTurk. We first present crowd workers a frame with a bounding box labeled for each object, and then ask them to draw the detailed segmentation for all objects within the bounding boxes. Each frame is labeled by three crowd workers and the    For NLC results are averaged over 12 videos as reported in their paper [9]. The best performing supervised and unsupervised methods are highlighted in bold. Metric: Jaccard score, higher is better. Please see appendix for per video results.
final segmentation is obtained by majority vote on each pixel.
The results indicate that our strategy to gather pseudo-ground truth is effective. On the 100 labeled frames, Jaccard overlap with the human-drawn ground truth is 77.8 (and 70.2 before pruning with bounding boxes).
Quantitative evaluation: We now present the quantitative comparisons of our method with several state-of-the-art methods and baselines, for each of the three datasets in turn. DAVIS dataset: Our method outperforms all existing video segmentation methods on this dataset and significantly advances state-of-the-art. Table I shows the results, with some of the best performing methods on this dataset taken from the benchmark results [49]. Our method is significantly better than simple flow baselines. This supports our claim that even though motion contains a strong signal about foreground objects in videos, it is not straightforward to simply threshold optical flow and obtain those segmentations. A data-driven approach that learns to identify motion patterns indicative of objects as opposed to backgrounds or camera motion is required. The appearance and motion variants of our method themselves result in a very good performance. The performance of the motion variant is particularly impressive, knowing that it has no information about object's appearance and purely relies on the flow signal. When combined together, the joint model results in a significant improvement, with an absolute gain of up to 11% over individual streams.
Our method is also significantly better than purely unsupervised methods, which typically rely on motion alone to identify foreground objects. This illustrates the benefits of a unified combination of both motion and appearance. Most surprisingly, our method significantly outperforms even the state-of-the-art semi supervised techniques, which require substantial human annotation on every video they process. The main motivation behind bringing a human in the loop is to achieve higher accuracies than fully automated methods, yet in this case, our proposed fully automatic method outperforms the best human-in-the-loop algorithms by a significant margin. For example, the BVS [30] method-which is the current best performing semi-supervised method and requires the first frame of the video to be manually segmented-achieves an overlap score of 66.5%. Our method significantly outperforms it with an overlap score of 71.51%, yet uses no human involvement. YouTube-Objects dataset: In Table II  NLC [9] FCP [20] Ours-Joint Ours-Joint that the appearance stream itself results in the overall best performance. Moreover, this dataset has a mix of static and moving objects which explains the relatively weaker performance of our motion model alone. The combined model works similarly well as appearance alone.
Overall, our method again outperforms the flow baselines and all the unsupervised methods by a significant margin (see Table II). Even among human-in-the-loop methods, we outperform all methods except IVID [19]. However we would like to point out that IVID [19] requires a human in the loop consistently to track the segmentation performance and correct whatever mistakes the algorithm makes. This can take up to minutes of human annotation time for each video. Our method uses zero human involvement but still performs very competitively.
Segtrack-v2 dataset: In Table III, our method outperforms all semi-supervised and unsupervised baselines except NLC [9] on Segtrack. While our approach significantly outperforms NLC [9] on the DAVIS dataset 4 , NLC is exceptionally strong on this dataset. Our relatively weaker performance could be due to the low quality and resolution of the Segtrack-v2 videos, making it hard for our network based model to process them. Nonetheless, our joint model still provides a significant boost over both our appearance and motion models, showing it again realizes the synergy of motion and appearance in a serious way. Qualitative evaluation: Fig. 4 shows qualitative results of our method. The top half shows visual comparisons between different components of our method including the appearance, motion, and joint models. We also show the optical flow image that was used as an input to the motion stream. These images help reveal the complexity of learned motion signals. In the bear example, the flow is most salient only on the bear's head, still our motion stream alone is able to segment the bear completely. The boat, car, and sail example shows that even when the flow is noisy-including strong flow on the background-our motion model is able to learn about object shapes and successfully suppresses the background regions. The rhino and train examples show cases where the appearance model fails to segment accurately but when combined with the motion stream, the joint model produces accurate segmentations.
The bottom half of Fig. 4 shows visual comparisons between our method and state-of-the-art unsupervised [8], [9] and semisupervised [20], [30] methods. The unsupervised methods have a very weak notion about object's appearance; hence they completely miss parts of objects [9] or cannot disambiguate the objects from background [8]. Semi-supervised methods [20], [30], which rely heavily on the initial human-segmented frame to learn about object's appearance, start to fail as time elapses and the object's appearance changes considerably. In contrast, our method successfully learns to combine generic cues about object motion and appearance, segmenting much more accurately across all frames even in very challenging videos.

V. CONCLUSIONS
We presented a new approach for learning to segment generic objects in video that 1) achieves deeper synergy between motion and appearance and 2) addresses practical challenges in training a deep network for video segmentation. Results show sizeable improvements over many existing methods-in some cases, even those requiring human intervention. In future work we plan to explore extensions that could permit individuation of multiple touching foreground objects, as well as ways to incorporate human intervention intelligently into our framework.

VI. APPENDIX
Per-video results for DAVIS and Segtrack-v2: Table IV shows the per video results for the 50 videos from the DAVIS dataset. Table I in the main paper summarizes these results over all 50 videos. We compare with several semi-supervised and unsupervised baselines. Our method outperforms the pervideo best unsupervised and best semi-supervised baseline in 25 out of 50 videos.    Table III in the main paper summarizes these results over all 14 videos. Our method outperforms several state-of-the art methods, including the ones which actually require human supervision during segmentation. For NLC results are averaged over 12 videos as reported in their paper [9]. The best performing supervised and unsupervised methods are highlighted in bold. Metric: Jaccard score, higher is better.