{'fraction_non_alphanumeric': 0.0365317990596387, 'fraction_numerical': 0.016023261568918583, 'mean_word_length': 5.113653555219365, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-supervised Video Object Segmentation by Motion Grouping

Animals have evolved highly functional visual systems to understand motion, assisting perception even under complex environments. In this paper, we work towards developing a computer vision system able to segment objects by exploiting motion cues, i.e. motion segmentation. We make the following contributions: First, we introduce a simple variant of the Transformer to segment optical flow frames into primary objects and the background. Second, we train the architecture in a self-supervised manner, i.e. without using any manual annotations. Third, we analyze several critical components of our method and conduct thorough ablation studies to validate their necessity. Fourth, we evaluate the proposed architecture on public benchmarks (DAVIS2016, SegTrackv2, and FBMS59). Despite using only optical flow as input, our approach achieves superior or comparable results to previous state-of-the-art self-supervised methods, while being an order of magnitude faster. We additionally evaluate on a challenging camouflage dataset (MoCA), significantly outperforming the other self-supervised approaches, and comparing favourably to the top supervised approach, highlighting the importance of motion cues, and the potential bias towards visual appearance in existing video segmentation models.


Introduction
When looking around the world, we effortlessly perceive a complex scene as a set of distinct objects. This phenomenon is referred to as perceptual grouping -the process of organizing the incoming visual information -and is usually considered a fundamental cognitive ability that enables understanding and interacting with the world efficiently. How do we accomplish such a remarkable perceptual achievement, given that the visual input is, in a sense, just a spatial distribution of variously colored individual points/pixels? In 1923, Wertheimer [80] first introduced the Gestalt principles with the goal of formulating the underlying causes by which sensory data is organized into groups, or Gestalten. The principles are much like heuristics with "a bag of tricks" [58] that the visual system may exploit for grouping, for example, proximity, similarity, closure, continuation, common fate, etc.
In computer vision, perceptual grouping is often closely related to the problem of segmentation, i.e. extracting the objects with arbitrary shape (pixel-wise labels) from clut-tered scenes. In the recent literature of semantic or instance segmentation, tremendous progress has been made by training deep neural networks on image or video datasets. While it is exciting to see machines with the ability to detect, segment, and classify objects in images or video frames, training such segmentation models through supervised learning requires massive human annotation, and consequently limiting their scalability. Even more importantly, the assumption that objects can be well-identified by their appearance alone in static frames is often an oversimplification -objects are not always visually distinguishable from their background environment. For instance when trying to discover camouflaged animals/objects from the background (Figure 1), extra cues, such as motion or sound, are usually required.
Among the numerous cues, motion is usually simple to obtain as it can be implicitly generated from unlabeled videos. In this paper, we aim to exploit such cues for object segmentation in a self-supervised manner, i.e. zero human annotation is required for training. At a high level, we aim to exploit the common fate principle, with the basic assumption being that elements tend to be perceived as a group if they move in the same direction at the same rate (have similar optical flow). Specifically, we tackle the problem by training a generative model that decomposes the optical flow into foreground (object) and background layers, describing each as a homogeneous field, with discontinuities occurring only between layers. We adopt a variant of the Transformer [72], with the self-attention being replaced by slot attention [44], where iterative grouping and binding have been built into the architecture. With some critical architectural changes, we show that pixels undergoing similar motion are grouped together and assigned to the same layer.
To summarize, we make the following contributions: first, we introduce a simple architecture for video object segmentation by exploiting motions, using only optical flow as input. Second, we propose a self-supervised proxy task that is used to train the architecture without any manual supervision. To validate these contributions, we conduct thorough ablation studies on the components that are key to the success of our architecture, such as a consistency loss on optical flow computed from various frame gaps. We evaluate the proposed architecture on public benchmarks (DAVIS2016 [55], SegTrackv2 [40], and FBMS59 [52]), outperforming previous state-of-theart self-supervised models, with comparable performance to the supervised approaches. Moreover, we also evaluate on a camouflage dataset (MoCA [39]), demonstrating a significant performance improvement over the other self-and supervised approaches, highlighting the importance of motion cues, and the potential bias towards visual appearance in existing video segmentation models.

Related Work
Video object segmentation has been a longstanding task in computer vision, which involves assigning pixels (or edges) of an image into groups (e.g. objects). In recent literature [4,9,11,15,23,24,30,33,34,37,38,47,50,51,53,54,56,56,70,73,74,75,76,84,87], two protocols have attracted increasing interest from the vision community, namely, semi-supervised video object segmentation (semi-supervised VOS), and unsupervised video object segmentation (unsupervised VOS). The former aims to re-localize one or multiple targets that are specified in the first frame of a video with pixel-wise masks, and the latter considers automatically segmenting the object of interest (usually the most salient one) from the background in a video sequence. Despite being called unsupervised VOS, in practice, the popular methods to address such problems extensively rely on supervised training, for example, by using two-stream networks [15,30,54,70] trained on large-scale external datasets. As an alternative, in this work, we consider a completely unsupervised approach, where no manual annotation is used for training whatsoever.
Motion segmentation shares some similarity with unsupervised VOS, but focuses on discovering moving objects. In the literature, [9,34,51,62,83] consider clustering the pixels with similar motion patterns; [15,69,70] train deep networks to map the motions to segmentation masks. Another line of work has tackled the problem by explicitly leveraging the independence of motion between the moving object and its background. For instance, [86] proposes an adversarial setting, where a generator is trained to produce masks, altering the input flow, such that the inpainter fails to estimate the missing information. In [5,6,39], the authors propose to highlight the independently moving object by compensating for the background motion, either by registering consecutive frames, or explicitly estimating camera motion. In constrained scenarios, such as autonomous driving, [60] proposes to jointly optimize depth, camera motion, optical flow and motion segmentation.
Optical flow computation is one of the fundamental tasks in computer vision. Deep learning methods allow efficient computation of optical flow, both in training on synthetic data [67,68], or learning with photometric loss in self-supervised [42,43] setting. In practise, flow has been useful for a wide range of problems, for example, pose estimation [18], representation learning [29,49], segmentation [9], and occasionally even used in lieu of appearance cues (RGB images) for tracking [61].
Transformer architectures have proven extremely adept at modelling long-term relationships within an input sequence via attention mechanisms. Originally used for language tasks [8,17,72], they have since been adapted to solve popular computer vision problems such as image classification [19], generation [13,59], video understanding [3,26,79], object detection [12], and zero-shot classification [57]. In this work, we take inspiration from a specific variant of self-attention, namely slot attention [44], which was demonstrated to be effective for learning objectcentric representations on synthetic data, e.g. CLEVR [31].
Layered representations were originally proposed by Wang and Adelson [77] to represent a video as a composition of layers with simpler motions. Since then, layered representations have been widely adopted in computer vision [7,32,36,85,90], often to estimate optical flow [65,66,81,82]. More recently, deep learning-based layer decomposition methods have been used to infer depth for novel view synthesis [64,88], separate reflections and other semi-transparent effects [1,2,25,45], or perform foreground/background estimation [25]. These works operate on RGB inputs and produce RGB layers, whereas we propose a layered decomposition of optical flow inputs for unsupervised moving object discovery.
Object-centric representations interpret scenes with "objects" as the basic building blocks (instead of individual pixels), which is considered an essential step towards humanlevel generalization. There is a rich literature on this topic, for example, IODINE [28] uses iterative variational inference to infer a set of latent variables recurrently, with each representing one object in an image. Similarly, MONet [10] and GENESIS [20] also adopt multiple encoding-decoding steps. In contrast, [44] proposes Slot Attention, which enables single step encoding-decoding with iterative attention. However, all works mentioned above have only shown applications for synthetic datasets, e.g. CLEVR [31]. In this paper, we are the first to demonstrate its use for object segmentation of realistic videos by exploiting motion, where the challenging nuances in visual appearance (e.g. the complex background textures) have been removed.

Method
Our goal is to take an input optical flow frame and predict a segment containing the moving object. We propose to train this model in a self-supervised manner, with an autoencoder-type framework. Specifically, our model outputs two layers: one representing the background, and the other for one or more moving objects in the foreground, as well as their oppacity layers (weighted masks). Formally: where I tÑt`n refers to the t to t`n input flow (backward flow when n ă 0), Φp¨q is the parametrized model, I i tÑt`n is the ith layer reconstruction, α i tÑt`n is its mask, and N " 2 is the number of layers (foreground and background). These layers can then be composited linearly to reconstruct the input image I tÑt`n :

Flow Segmentation Architecture
For simplicity, we first consider the case of a single flow field as input (depicted in the top part of Figure 2). The entire model consists of three components: (1) a CNN encoder to extract a compact feature representation, (2) an iterative binding module with learnable queries that plays a similar role as soft clustering, i.e. assigning each pixel to one of motion groups, and (3) a CNN decoder that individually decodes each query to full resolution layer outputs (where thresholding the alpha channel yields the predicted segment).
CNN encoder. We first pass the precomputed optical flow between two frames, I tÑt`n P R 3ˆH0ˆW0 , to a CNN encoder Φ enc , which outputs a lower-resolution feature map: where H 0 , W 0 and H, W refer to the spatial dimensions of the input and output feature maps respectively. Note that, we convert the flow into a three-channel image using the traditional method in the optical flow literature [67].
Iterative binding. The iterative binding module Φ bind aims to group image regions into single entities based on their similarities in motion, i.e. pixels moving in the same direction at the same rate should be grouped together. Intuitively, such a binding process requires a data-dependent parameter updating mechanism, iteratively enriching the model, gradually including more pixels undergoing similar motions.
To accomplish this task, we adopt a simple variant of slot attention [44], where instead of Gaussian-initialized slots, we use learnable query vectors. Slot attention has recently shown remarkable performance for object-centric representation learning, where the query vectors compete to explain parts of the inputs via a softmax-based attention mechanism, and the representations of these slots are iteratively updated with a recurrent update function. In our case of motion segmentation, ideally, the final representation in each query vector separately encodes the moving object or the background, which can then be decoded and combined to reconstruct the input flow fields.
Formally, our inputs to Φ bind are feature maps F tÑt`n and two learnable queries (representing foreground and background) Q P R Dˆ2 . Learnable spatial positional encodings are summed with F tÑt`n ; with some abuse of notation, we still refer to this sum as F tÑt`n . We use three different linear transformations to generate the query , key and value: q P R Dˆ2 , k, v P R DˆHW , where In contrast to the standard Transformer [72], the coefficients in slot attention are normalized over all slots. This choice of normalization introduces competition between the slots to explain parts of the input, and ensures each pixel is assigned to a query vector: To aggregate the input values to their assigned query slot, a weighted mean is used as follows: where, A i,j :" To maintain a smooth update of the query slots Q, the aggregated vectors U are fed into a recurrent function, parametrized with Gated Recurrent Units (GRU), This whole binding process is then iterated T times. The pseudocode can be found in the Supplementary Material. CNN decoder. The CNN decoder Φ dec individually decodes each of the slots to outputs of original resolution (tÎ i tÑt`n , α i tÑt`n u P R 4ˆH0ˆW0 ), which includes an (unnormalized) single-channel alpha mask and the reconstructed flow fields. Specifically, the input to the decoder is the slot vector broadcasted onto a 2D grid augmented with a learnable spatial positional encoding. Reconstruction. Once each slot has been decoded, we apply softmax to the alpha masks across the slot dimension, and use them as mixture weights to obtain the reconstruc-tionÎ tÑt`n (Eq. 2). Our reconstruction loss is an L2 loss between the input and reconstructed flow, where p is the pixel index, and Ω is the entire spatial grid.
Entropy regularization. We impose a pixel-wise entropy regularisation on inferred masks: This loss is zero when the alpha channels are one-hot, and maximum when they are of equal probability. Intuitively, this helps encourage the masks to be binary, which aligns with our goal in obtaining segmentation masks.
Instance normalisation. In the case of motion segmentation, objects can only be detected if they undergo an independent motion from the camera; thus previous work attempts to compensate for camera motion [5,39]. We are inspired by these ideas, but instead of explicitly estimating homography or camera motion, we take a poor-man's approach by simply using Instance Normalisation (IN) [71] in the CNN encoder and decoder, which normalizes each channel of the training sample independently. Intuitively, the mean activation tends to be dominated by the motions in the large homogeneous region, which is usually the background. This normalization, in combination with ReLU activations, helps gradually separate the background motion from the foreground motions. This is experimentally shown in Section 5.1

Self-supervised Temporal Consistency Loss
The segmentation computed for the current frame should be identical irrespective of whether the 'second' frame is consecutive, or earlier or later in time. We harness this constraint to form a self-supervised temporal consistency loss by first defining a set of 'second' frames, and then requiring consistency between their pairwise predictions. We describe the set first, followed by the loss.

Multi-step flow.
As objects may be static for some frames, we make our predictions more robust by leveraging observations from multiple timesteps. We consider the flow fields computed from various temporal gaps as an input set, i.e. tI tÑt`n1 , I tÑt`n2 u, n 1 , n 2 P t´2,´1, 1, 2u, and use a permutation invariant consistency loss to encourage the model to predict the same foreground/background segmentation for all flow fields in the set.
Consistency loss. We randomly sample two flow fields from the input set and pass them through the model (Φp¨q), outputting the flow reconstruction and alpha masks for each. As the reconstruction loss is commutative, it is not guaranteed that the same slot will always output the background layer; therefore, we use a permutation-invariant consistency loss, i.e. only backpropagating through the lowest-error permutation: Note that, this consistency enforcement only occurs during training. At inference time, a single-step flow is used, as shown in the top half of Figure 2.
Total loss. The total loss for training the architecture is: we use γ r " 10 2 , γ c " 10´2 and γ e " 10´2, but we found the model to be fairly robust to these hyperparameters.

Discussion
Differences from slot attention. Slot attention was originally introduced for self-supervised object segmentation for RGB images [44], and its usefulness was demonstrated on synthetic data (CLEVR [31]), where objects are made of primitive shapes with simple textures. However, this assumption is unlikely to hold in the case of natural images or videos, making it challenging to generalize such objectcentric representations.
In this work, we build on the insight that although objects in images may not be naturally textureless, their motions typically are. Hence, we develop the self-supervised object segmentation model by exploiting their optical flows, where the nuance in visual appearance is discarded, thus not restricted to simple synthetic cases. As an initial trial, we experimented with the same setting as [44], where query vectors are sampled from a Gaussian distribution; however, we were unable to train it. Instead, we use learnable embeddings here, which we highlight as one of the architectural changes critical to our model's success. Other critical changes include instance normalization and temporal consistency, which we demonstrate in ablations in Section 5.1. Why does it work for motion segmentation? The proposed idea can be seen as training a generative model to segment the flow fields. With the layered formulation, reconstruction is limited to be a simple linear composition of layer-wise flow, decoded from a single slot vector.
Conceptually, this design has effectively introduced a representational bottleneck, encouraging each slot vector to represent minimal information, i.e. homogeneous motion, and with minimal redundancy (mutual information) between slots. All these properties make such an architecture well-suited to the task of segmenting objects undergoing independent motions.

Datasets
DAVIS2016 [55] contains a total of 50 sequences (30 for training and 20 for validation), depicting diverse moving objects such as animals, people, and cars. The dataset contains 3455 1080p frames with pixel-wise annotations at 480p for the predominantly moving object.
FBMS59 [52] consists of 59 sequences and 720 annotated frames (every 20th frame is annotated), which vary greatly in image resolution. Sequences involve multiple moving objects, some of which may be static for periods of time.
Moving Camouflaged Animals (MoCA) [39] contains 141 HD video sequences, depicting 67 kinds of camouflaged an-imals moving in natural scenes. Both temporal and spatial annotations are provided in the form of tight bounding boxes for every 5th frame. Using the provided motion labels (locomotion, deformation, static), we filter out videos with predominantly no locomotion, resulting in 88 video sequences and 4803 frames.

Evaluation Metrics
Segmentation (Jaccard). For DAVIS2016, SegTrackv2 and FBMS59, pixelwise segmentation is provided; thus we report the standard metric, region similarity (J ), computing the mean over the test set. For FBMS59 and SegTrackv2, we follow the common practice [30,86] and combine multiple objects as one single foreground.
Localization (Jaccard & Success Rate). As the MoCA dataset provides only bounding box annotations, we evaluate for the detection task and report results in the form of detection success rate [21,41], for varying IoU thresholds (τ P t0.5, 0.6, 0.7, 0.8, 0.9uq.

Implementation Details
We evaluate three different approaches for computing optical flow, namely, PWC-Net [67], RAFT [68] and ARFlow [42]; the first two are supervised, while the latter is self-supervised. We extract the optical flow at the original resolution of the image pairs, with the frame gaps n P t´2,´1, 1, 2u for all datasets, except for FBMS59, where we use n P t´6,´3, 3, 6u to compensate for small motion. To generate inputs to the network for training, the flows are resized to 128ˆ224 (and scaled accordingly), converted to 3-channel images with the standard visualization used for optical flow, and normalized to r´1, 1s.
In the iterative binding module (Φ bind p¨q), we use two learnable query vectors (as we consider the case of segmenting a single moving object from the background), and choose T " 5 iterations (as explained in Section 3.1). We adopt a simple VGG-style network for the CNN encoder and decoder with instance normalization. We train with a batch size of 64 images and use the Adam optimizer [35] with an initial learning rate of 5ˆ10´4, decreasing every 80k iterations. The exact architecture description and training schedule can be found in the Supplementary Material.

Results
In this section, we compare primarily with a topperforming approach trained without manual annotations -Contextual Information Separation (CIS [86]). However, as the architecture, input resolution, modality and postprocessing are all different, we try our best to conduct the comparison as fairly as possible. Note that benchmarks are evaluated at full resolution by simply upsampling the predicted masks.

Ablation Studies
We conduct all ablation studies on DAVIS2016, and vary one variable each time, as shown in Table 1.
Choice of optical flow algorithm. With the same flow extraction method (PWC-Net), our proposed model (Ours-A) outperforms CIS by about 4.5 points on mean Jaccard (J ), and using improved optical flow (RAFT) provides further performance gains. We therefore use RAFT from hereon.
Instance normalization and grouping. We observe two phenomena: first, when holding constant on the number of grouping iterations T (3 or 5), models trained with instance normalization perform consistently better; second, iterative grouping with T " 5 is better than that trained with T " 3. However, at T " 8, the model did not converge in the same number of training steps, and thus we do not include it in the table. For the remainder of the experiments, we use instance normalization and T " 5.
Consistency and entropy regularization. While comparing Ours-B and Ours-I, we observe that the performance degrades significantly without the temporal consistency loss, and that the entropy regularization is also important, as shown by Ours-B and Ours-H.

Comparison with State-of-the-art
We show our results in Table 2. On DAVIS2016, we improve upon the state-of-the-art for unsupervised methods (CIS) by a large margin (`9.1%). As shown in Figure 3, despite not using any pixel-level annotations during training, our method is nearing the performance of supervised models trained on thousands of images.
In addition, we argue that, motion segmentation in realistic scenarios, e.g. by predator or prey, is likely to require fast processing. Our model operates on small resolution (potentially sacrificing some accuracy) with over 80fps. Our method's efficiency gain mainly comes from two sources: first, our model is a lightweight VGG-style network with only 4.77M parameters; second, we disregard any postprocessing used in previous approaches, e.g. averaging the prediction across multiple flow steps, across multiple crops, temporal smoothing, or CRFs, which cost over 10s in total.
For SegTrackv2 and FBMS59, they occasionally include multiple objects in a single video, and only a subset of them are moving, making it challenging to spot all objects using flow-only input, but we achieve competitive performance nonetheless. We discuss this limitation below.

Camouflage Breaking
In addition, we also benchmark the model on camou   approaches may use ImageNet pretraining [16], but here we only count images with pixel-wise annotations.  Table 2: Full comparison on moving object segmentation (unsupervised video segmentation). We consider three popular datasets, DAVIS2016, SegTrack-v2 (STv2), and FBMS59. Models above the horizontal dividing line are trained without using any manual annotation, while models below require ground truth annotations at training time. Numbers in parentheses denote the additional usage of significant post-processing, e.g. multi-step flow, multi-crop, temporal smoothing, CRFs. Runtime excludes optical flow computation.
self-supervised manner. We convert the output segmentation mask into a bounding box by drawing a bounding box around the largest connected region in the predicted mask.
We report quantitative results in Table 3 and show qualitative results in Figure 4. Our model significantly outperforms CIS (14% when allowing no post-processing), previous supervised approaches e.g. COD [39] (18.5% on Jaccard), and even COSNet [46] (among the top supervised approaches on DAVIS). We conjecture that COSNet's weaker performance is due to its sole reliance on visual appearance (which is distracting for the MoCA data) rather than using motion inputs. This is particularly interesting, as it clearly indicates that no single information cue is able to do the task perfectly, echoing the two-stream hypothesis [27] that both appearance and motion are essential to visual systems.

Limitations
Despite showing remarkable improvements on motion segmentation in accuracy and runtime, we note the following limitations of the proposed approach (shown in Figure 4) and treat them as future work: first, the existing benchmarks are mostly limited to motion segmentation into foreground and background, thus, we choose to use two slots in this paper; however, in real scenarios, videos may contain multiple independently moving objects, which the current model will assign to a single layer. It may be desirable to further separate these objects into different layers. Second, we have only explored motion (optical flow) Figure 4: Qualitative results. On DAVIS2016 (left), our method is able to segment a variety of challenging objects, often on-par with top supervised approaches. On MoCA (right), our model is able to accurately segment well-camouflaged objects even when previous supervised methods fail completely (3rd, 4th columns). We show a failure case (left) where the splash created by the person is incorrectly included in our predicted segment, and another failure case (right) where the animal is only partially moving and thus partially segmented.
Both CIS and Ours were pre-trained on DAVIS and finetuned on MoCA in a self-supervised manner. Our method achieves comparable Jaccard (J ) to MATNet (2nd best model on DAVIS), without using RGB inputs and without any manual annotation for training.
as input, which significantly limits the model in segmenting objects when flow is uninformative or incomplete (as in Figure 4, right); however, the self-supervised video object segmentation objective is applicable also to a two-stream approach, and so RGB could be incorporated. Third, the current method may fail when optical flow is noisy or lowquality (Figure 4, left); jointly optimizing flow and segmentation is a possible way forward in this case.

Conclusion
In this paper, we present a self-supervised model for motion segmentation. The algorithm takes only flow as input, and is trained without any manual annotation, surpassing previous self-supervised methods on public benchmarks such as DAVIS2016, narrowing the gap with su-pervised methods. On the more challenging camouflage dataset (MoCA), our model actually compares favourably to the top approaches in video object segmentation that are trained with heavy supervision. As computation power grows and more high quality videos become available, we believe that self-supervised learning algorithms can serve as a strong competitor to the supervised counterparts for their scalability and generalizability.

A. Training Details
In this section, we include the details for reproducing our results, for example, architectures, pseudo-codes and hyperparameters.

A.1. Encoder & Decoder
The backbone of network architecture for the model are shown in Table 4 Table 4: Network architecture. All convolutions have padding 2 to preserve spatial resolution, and are followed by instance normalization and ReLU activation, except the final layer. The details of the iterative binding module is in Figure 5.

A.2. Iterative Binding Module
The pseudocode for the iterative binding module is shown in Figure 5. The full code is available in the submitted source code, and will be made publicly available.

A.3. Hyperparameters
For all datasets, we train with batch size of 64, although note that this corresponds to 32 pairs of optical flow in order to train with consistency loss. Our initial learning rate is 5e-4, with first 200 steps being warmup, and decays by half every 8e4 iterations. During this decay, the scale for entropy and consistency loss is also increased by a factor of 5, gradually encouraging the predicted alpha channel to be binary. We train the algorithm for about 300k iterations.

B. MoCA dataset curation
The MoCA dataset contains 141 high-definition video sequences, with an average duration of 11 seconds. These sequences were collected from YouTube with resolution 720ˆ1280, and sampled at 24 fps, resulting in 37K frames depicting 67 kinds of camouflaged animals moving in natural scenes. Both temporal and spatial annotations are provided in the form of tight bounding boxes on every 5th frame. We use a modified version of this dataset in order to make it more suitable for segmentation tasks. We outline the process and rationale below.
• We crop away the channel logos and empty border spaces, and resize the low-resolution images to the same resolution as the other images in the dataset (at 720ˆ1280). We then adjust the ground-truth annotations accordingly.
• The original authors resampled all videos to 24 fps even when some original videos have less, causing some consecutive frames to be identical. To alleviate this, we sample every 3 frames from the original dataset, up to 100 frames per video.
• For annotations, we use linear interpolation to generate the missing frames' bounding boxes, resulting in a dense frame-wise annotation.
• The authors provided the motion labels for each annotated frame (locomotion, deformation, static), so we filter away videos with predominantly no locomotion.
• We also further discard videos that contain large amount of frames where the motion does not belong to the primary object.
This eventually results in 88 video sequences and 4803 frames, which we will release for fair comparison.

C. Results breakdown
The main evaluation metric used in this paper is the Jaccard score (J ), which is the intersection-over-union between the predicted and ground-truth masks. In line with previous works, we show the per-category results breakdown for our model on DAVIS2016, SegTrackv2, FBMS59, and MoCA in Tables 5-8. Note that, since we focus on both speed and accuracy, the predictions are only of 128ˆ224 pixels, and we directly upsample this prediction to the original resolution and compare with the groundtruth.

D. Qualitative results
We show more qualitative results in Figures 6 and 7