{'fraction_non_alphanumeric': 0.04225655612299869, 'fraction_numerical': 0.016441485120779194, 'mean_word_length': 4.838953195772522, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Joint Anchor-Feature Refinement for Real-Time Accurate Object Detection in Images and Videos

Object detection has been vigorously investigated for years but fast accurate detection for real-world scenes remains a very challenging problem. Overcoming drawbacks of single-stage detectors, we take aim at precisely detecting objects for static and temporal scenes in real time. Firstly, as a dual refinement mechanism, a novel anchor-offset detection is designed, which includes an anchor refinement, a feature location refinement, and a deformable detection head. This new detection mode is able to simultaneously perform two-step regression and capture accurate object features. Based on the anchor-offset detection, a dual refinement network (DRNet) is developed for high-performance static detection, where a multi-deformable head is further designed to leverage contextual information for describing objects. As for temporal detection in videos, temporal refinement networks (TRNet) and temporal dual refinement networks (TDRNet) are developed by propagating the refinement information across time. We also propose a soft refinement strategy to temporally match object motion with the previous refinement. Our proposed methods are evaluated on PASCAL VOC, COCO, and ImageNet VID datasets. Extensive comparisons on static and temporal detection verify the superiority of DRNet, TRNet, and TDRNet. Consequently, our developed approaches run in a fairly fast speed, and in the meantime achieve a significantly enhanced detection accuracy, i.e., 84.4% mAP on VOC 2007, 83.6% mAP on VOC 2012, 69.4% mAP on VID 2017, and 42.4% AP on COCO. Ultimately, producing encouraging results, our methods are applied to online underwater object detection and grasping with an autonomous system. Codes are publicly available at https://github.com/SeanChenxy/TDRN.


I. INTRODUCTION
Object detection is one of the fundamental and challenging areas of research in computer vision. With rapid advances in deep learning, convolutional neural networks (CNN) have demonstrated the state-of-the-art performance in this task. Zhao et al. presented an overview of modern object detection approaches [1]. From this review, we can see that two-stage detectors represented by RCNN family [2] and RFCN [3] usually attain an accurate yet slightly slow performance. On the contrary, by detecting objects in a one-step fashion, singlestage detectors [4], [5] are able to run in real time with reasonably modest accuracy. Therefore, fast accurate detection remains a challenging problem for real-world applications.
It is instructive that the two-stage method induces high accuracy, while the single-stage detector has a desirable inference speed. This inspires us to investigate the reasons. In our opinion, the high accuracy of two-stage approaches comes with two advantages: i) two-step regression and ii) relatively accurate features for detection. In detail, two-stage detectors firstly regress pre-defined anchors with the aid of region proposal [2], and this operation significantly eases the difficulty of final localization. Besides, an RoI-wise subnetwork [2] is appended to the region proposal part, so features in region of interest can be leveraged for final detection. By contrast, there are two drawbacks in the single-stage paradigm: i) Detection head directly regresses coordinates from pre-defined anchors, but most anchors are far from matching object regions. ii) Classification information comes from probably inaccurate locations, where features could not be precise enough to describe objects. Referring to Fig. 1(a), it is relatively difficult to regress pre-defined anchors to precisely surround the object (e.g., the dog in Fig. 1). Moreover, as feature sampling locations follow pre-defined anchor regions, detection features for small-scale anchors cannot cover the entire object region, while that for large-scale anchors weaken the object because of background. On the contrary, owing to region proposal, the two-stage methods detect the dog using a better initialization (see Fig. 1(b)). Thus, the strengths of two-stage methods exactly reflect the single-stage drawbacks that lead to relatively lower detection accuracy. Although Zhang et al. developed RefineDet [6] to introduce two-step regression to the singlestage detector, it still failed to capture accurate detection features. That is, pre-defined feature sampling locations are not precise enough for describing refined anchor regions. (Note that detailed comparison between RefineDet and our approach will be presented in Section III-B.) Thus, there is an imperative need of further overcoming these single-stage limitations for real-time accurate object detection.
In addition, most researches have largely focused on detecting object statically, ignoring temporal coherence in real-world applications. Detection in real-world scenes was introduced by ImageNet video detection (VID) dataset [18]. To the best of our knowledge, main ideas of temporal detection include i) post-processing [7], ii) tracking-based location [8], [9], iii) feature aggregation with motion information [8], [10]- [13], iv) RNN-based feature propagation [12], [14]- [16], and v) batch-frame processing (i.e., tubelets proposal) [17]. All these ideas are attractive in that they are able to leverage temporal information for detection, but they also have respective limitations. In brief, methods i)-iv) borrow other tools (e.g., tracker, optical flow, LSTM, etc.) for temporal analysis. Methods iii) and iv) focus on constructing superior temporal features. Nevertheless, they detect objects following the static mode. Method v) works in a non-causal offline mode that prohibits this approach from real-world tasks. Furthermore, most recent works pay excessive attention to accuracy so that high computational costs could affect time efficiency. Thus, a novel temporal detection mode should be developed for videos.
Overcoming aforementioned single-stage drawbacks, a dual refinement mechanism is proposed in this paper for static and temporal visual detection, namely anchor-offset detection. This joint anchor-feature refinement includes an anchor refinement, a feature location refinement, and a deformable detection head. The anchor refinement is developed for two-step regression, while the feature location refinement is proposed to capture accurate single-stage features. Besides, a deformable detection head is designed to leverage this dual refinement information. Based on the anchor-offset detection, we propose three approaches for object detection in images and videos. Firstly, a dual refinement network (DRNet) is proposed. DRNet is designed for static detection, where a multi-deformable head is developed for diversifying detection receptive fields for more contextual information. Secondly, temporal refinement networks (TRNet) are designed, which perform anchor refinement across time for video detection. Thirdly, temporal dual refinement networks (TDRNet) are developed that extend the anchor-offset detection towards temporal tasks. Additionally, for temporal detection task, we propose a soft refinement strategy to match object motion with previous refinement information. Our proposed DRNet, TRNet, and TDRNet are validated on PASCAL VOC [19], COCO [20], and ImageNet VID [18] datasets. As a result, our methods achieve a real-time inference speed and considerably improved detection accuracy. Furthermore, these approaches have been applied to objectdriven navigation and grasping in an unstructured undersea environment. Contributions are summarized as follows: • Starting with drawbacks of single-stage detectors, an anchor-offset detection is proposed to perform two-step regression and capture accurate object features. The anchor-offset detection includes an anchor refinement, a feature-offset refinement, and a deformable detection head. Academically, without region-level processing, this joint anchor-feature refinement achieves single-stage region proposal. Thus, the anchor-offset detection bridges single-stage and two-stage detection so that it is able to induce a new detection mode. • A DRNet based on the anchor-offset detection and a multi-deformable head is developed to elevate static detection accuracy while maintaining real-time inference speed for image detection task. • As a new temporal detection mode for video detection task, a TRNet and a TDRNet are proposed based on the anchor-offset detection without the aid of any other temporal modules. They are characterized by a better accuracy vs. speed trade-off and have a concise training process without the requirement of sequential data. In addition, a soft refinement strategy is designed to enhance the effectiveness of refinement information across time. and TDRNet obtains 67.3% mAP. The remainder of this paper is organized as follows. Section II presents the related works. Including anchor-offset detection and multi-deformable head, DRNet is elaborated in Section III. Section IV presents TRNet and TDRNet in detail, and Section V provides the experimental results and discussion. Conclusions are summarized in Section VI.

A. CNN-Based Static Object Detection
Deep learning methods have recently dominated the field of object detection [1]. Two-stage detectors [2], [3], [21], [22] usually detect objects by region proposal, location, and classification. For example, inspired by Faster RCNN [2] and RFCN [3], CoupleNet [21] leveraged both region-level and part-level features to express a variety of challenging object situations, which achieved considerable detection accuracy but it only ran at 8.2 FPS. As groundbreaking works, YOLO [4] and SSD [5] localized and classified objects using a singleshot network for real-time detection. Recently, many revised single-stage versions have emerged [6], [24]- [29]. Typically, in favor of small object detection, Lin et al. developed a RetinaNet to formulate the single-shot network as an FPN [23] fashion for propagating information in a top-down manner to enlarge shallow layers' receptive field [24]. Redmon and Farhadi proposed YOLOv3 with DarkNet53 and multi-scale anchor for fast accurate detection [29]. Zhang et al. designed a RefineDet to introduce two-step regression to single-stage pipeline [6]. RefineDet adjusted pre-defined anchors for more  Fig. 2. The schematic layout of the proposed DRNet. Refined anchors are produced by coarse regression with ARM features, and they are first employed to predict feature offsets, namely, feature location refinement. The detection head utilizes ODM feature maps, refined anchors, and refined feature sampling locations to detect objects, i.e., anchor-offset detection. A multi-deformable head is designed for rich contextual information. (a) Overall framework. Each anchor refinement induces a feature location refinement. (b) Design details. Convolution is detailed with kernel size, stride, and output channel size. Only one detection path with 3 × 3 convolutional kernel is shown. precise localization. However, its detection features were still fixed on pre-defined positions, failing to precisely describe refined anchor regions. In short, although single-stage methods has a superiority in speed, two-stage methods still dominate detection accuracy on generic benchmarks [18]- [20]. Hence, we are motivated to analyze single-stage drawbacks from twostage merits (analyzed in Section I), and construct DRNet with both competitive accuracy and fast speed.

B. Temporal Object Detection
To detect objects in temporal vision, some post-processing methods have been first investigated to merge multi-frame results, then tracker-based detection, motion-guided feature aggregation, RNN-based feature integration, and tubelet proposal are studied by the research community. Han et al. proposed an SeqNMS to discarded temporally interrupted bounding boxes in the non-maximum suppression (NMS) phrase [7]; Feichtenhofer et al. combined RFCN and a correlation-filter-based tracker to boost recall rate [9]. Based on motion estimation with optical flow, Zhu et al. devised a temporally-adaptive key frame scheduling to effective feature aggregation [11]; Chen et al. and Liu et al. took advantage of Long Short-Term Memory to propagate CNN features across time [15], [16]. However, the temporal analysis capacity in the abovementioned methods is obtained from other temporal tools. Although some methods focused on how to construct superior temporal features, they still remained inapposite static detection mode. As a typical offline detection mode, Kang et al. reported a TPN for tubelet proposal (i.e., temporally propagated boxes) so that multiple frames could be simultaneously processed to improve temporal consistency [17]. However, this batch-frame mode struggled to be qualified for real-world tasks. On the contrary, without the aid of any other temporal tools, we novelly develop a real-time online detection mode for videos using the idea of refinement. That is, refined anchors and refined feature sampling locations are generated with key frames, which would be temporally propagated for detection. Compared to most video detectors, our methods have a concise training process without the need for sequential images.

C. Sampling for Detection
It is widely accepted that spatial sampling is important to construct robust features. For example, Peng et al. detected objects by an improved multi-stage particle window that can sample a small number of key features for detection [34]. In terms of CNN, canonical convolution is based on a square kernel that is not suited enough to variform objects. For augmenting the spatial sampling locations, Dai et al. proposed deformable convolutional networks to combat fixed geometric structures in traditional convolution operation. The deformable convolution significantly boosted the detection accuracy of RFCN [35]. As for video detection, Bertasius et al. used the deformable convolutions across time and constructed robust features for temporally describing objects [13]. Zhang et al. designed a feature consistency module with deformable convolution to reduce inconsistency in the single-stage pipeline [36]. Wang et al. proposed guided anchoring for RPN, Faster RCNN, and RetinaNet to achieve higher-quality region proposal [37]. Creatively, we tend to capture accurate single-stage features, and more specifically, refined feature locations are produced based on refined anchors. Moreover, we propagate refinement information across time for video detection.

III. DUAL REFINEMENT NETWORK
In this section, the proposed DRNet will be presented. The network architecture is first briefed, then, we will demonstrate how to overcome two key single-stage drawbacks with anchoroffset detection. Next, our designed multi-deformable head is delineated, followed by the training and inference.
A. Overall Architecture 1) Basic Structure: As shown in Fig. 2, our proposed architecture is a single-shot network with a forward backbone for feature extraction. The network generates a fixed number of bounding boxes and corresponding classification scores, followed by the NMS for duplicate removal. Inheriting from Re-fineDet [6], there is an anchor refinement module (ARM) and an object detection module (ODM) for two-step regression. ARM regresses coordinates for refined anchors, then feature offsets are predicted using anchor offsets. In ODM, a creative detection head is designed with deformable convolution for final classification and regression, whose inputs are ODM features, refined anchors, and feature offsets. Furthermore, a multi-deformable head is developed with multiple detection paths to leverage contextual information for detection.
B. Anchor-Offset Detection 1) From SSD to RefineDet, then to DRNet: As illustrated in Fig. 3(a), SSD directly detects objects with ARM features, whereas RefineDet adopts FPN for strong semantic information. Moreover, RefineDet develops an anchor refinement for more precision localization and a negative anchor filtering for addressing extreme class imbalance problem. In our DR-Net, we inherit anchor refinement but discard the negative anchor filtering since training with hard negative mining [5] induces a similar effect. More specifically, a feature location refinement and a deformable detection head are proposed to combat another key drawback in the single-stage paradigm, i.e., inaccurate feature sampling locations.
In general, detection in traditional SSD-like manner is based on hand-crafted anchors which are rigid and usually inaccurate. Pre-defined anchors and fixed feature locations could not be suited enough to regress and classify objects (see the left top in Fig. 3(b)). Through preliminary localization, refined anchors in RefineDet are in favor of more precise coordinate prediction. However, RefineDet still uses inaccurate feature sampling locations (see the right top in Fig. 3(b)) for regression and classification. That is, the anchor refinement would incur serious anchor-feature misalignment. Thus, it is defective that the anchor refinement is leveraged alone. Overcoming these difficulties, our designed anchor-offset detection is able to achieve two-step regression and capture more accurate detection features in a single-stage pipeline (see the left bottom in Fig. 3(b)). This joint anchor-feature refinement manner is more reasonable than RefineDet.
2) Anchor Refinement: This process is analogous to Re-fineDet, i.e., ARM generates refined anchors that provide better initialization for the second-step regression. A location head performs convolution to generate anchor offset ar using backbone-based ARM features f ARM . That is, ar = W ar * f ARM , where * denotes convolution (W is the learnable convolutional weight). Note that ar is the coordinate offset from original anchors. Anchor refinement is urgently necessary. On one hand, it highly relieves the difficulty of localization. On the other hand, it can guide feature location refinement.
3) Deformable Detection Head: According to deformable convolution [35], a deformable detection head is designed to leverage the refinement information. The standard detection head in SSD uses a regular 3 × 3 grid R to predict category probability and coordinates for a feature map cell. In the meantime, through careful anchor design, the respective field of R can describe a specific anchor region. Thus, the prediction can be given as P p0 = p∈R w(p) · f ODM (p), where P is the prediction of category probability or coordinate offset; w is the convolution weight; p represents positions in R while p 0 is the center; f ODM denotes ODM features.
However, the respective field of R usually fails to describe the refined anchor region (see the right top of Fig. 3(b)). Thereby, allowing R to deform to fit various anchor changes, the deformable detection head is developed to capture accurate features with the feature offset δp, (1)

4) Feature Location Refinement:
The offset set ∆p = {δp} is computed with the input feature in original deform pipeline, where W f r is the convolutional weight. Nevertheless, there is a strong demand for describing the refined anchor regions with the deformed grids. Therefore, our feature offsets are predicted based on anchor offsets, i.e., feature location refinement, In detail, this operation is a convolution with 1 × 1 kernels. Since each spatial element in ar is coordinate offsets for refined anchors, its channel information is fused for feature location refinement. Note that anchor offsets and feature offsets are different in tensor shape. A man-made function can be designed to map anchor offsets to feature offsets, but we adopt a learnable mapping. Although the man-made manner can also promote refined feature locations to describe refined anchors, it still generates a conventional regular feature sampling region. Thus, the feature location refinement adopts a learnable manner to produce flexible feature sampling locations from multiple anchor offsets.
In this way, the refined feature locations can describe refined anchor regions more effectively. We call this detection mode anchor-offset detection, which can be formulated as where ⊕, ao represent anchor decoding operation [5] and the original anchor, respectively; W * (f, ∆p) denotes deformable convolution with W as the weight. As ar is the coordinate offset from ao, ar ⊕ ao is the refined anchor. The operation of two ⊕ is two-step regression that elevates the precision of localization, while ∆p is the feature offset that constructs the accurate single-stage detection features.

C. Multi-Deformable Head
CoupleNet developed local and global FCN to detect objects [21]. The local FCN focused local features in a region proposal while the global one paid attention to the whole region-level features. In this way, more semantic information and underlying object relation are exploited for high-quality detection. Thus, taking aim at describing the object using original, shrunken, and expansile region-level features, a multideformable head is developed for the single-stage detector. The shrunken region-level features are in favor of leveraging local messages while the expansile region-level features contain more contextual information and object relation.
In this way, multiple detection head is designed with different respective field sizes, inducing multiple detection paths. As shown in Fig. 4, each of detection path is an anchor-offset detection, and their feature location refinement is independent. Besides, their results are fused with element-wise summation. The detection based on L deformable paths can be given as

D. Training and Inference
As for pre-defined anchor setting, each feature layer is associated with one specific scale of anchors, i.e., the anchor size of [32,64,128,256] is adopted for 4-scale feature maps from low-level to high-level, and 3 anchors are tiled at each feature map cell with aspect ratios of [1.0, 2.0, 0.5]. In terms of optimization, an SGD optimizer with 0.9 momentum and 0.0005 weight decay is employed to train the whole network. Because of different sizes of datasets, the learning rate schedule is diverse for each dataset, which will be briefed latter.
A multi-task objective is designed to train DRNet including two localization losses L loc−ARM , L loc−ODM and a confidence loss , where g * i is the ground truth coordinates of the i-th positive anchor. Before computation of L loc , anchors should be determined to be positive or negative based on jaccard overlap [5]. We handle original anchors and refined anchors for L loc−ARM and L loc−ODM , respectively, by the following processes. Firstly, each ground truth box is matched to anchors with the best jaccard overlap, then anchors with > 0.5 overlap will be matched to corresponding ground truth box. Let c cls i be the probability that the i-th predicted box belongs to class cls (cls = 0 for background).
where δN negative anchors are selected by hard negative mining [5]. This operation selects a part of negative boxes with top loss values for training to address the problem with extreme foregroundbackground class imbalance, and δ = 3.
In inference phase, DRNet predicts confident object candidates (confident scores > 0.01) in the manner of anchoroffset detection and multi-deformable head. Subsequently, these candidates are processed by NMS with 0.45 jaccard overlap pre class and retain top 200 (for COCO) or 300 (for VOC and VID) high confident objects as the final detections.

IV. TEMPORAL DUAL REFINEMENT NETWORKS
In this section, we present how to propagate refined anchors and refined feature sampling locations across time. Next, TRNet and TDRNet are formed. Then, we also describe the proposed soft refinement strategy.

A. Architecture
As shown in Fig. 5(a), a reference generator (RG) and a refinement detector (RD) are designed in this section, both of which are constructed with similar structure, i.e., canonical SSD framework [5] with 4-scale detection features. However, RG and RD have different training mode, parameters, and outputs. Like ARM in DRNet, RG predicts refinement information including refined anchors or both refine anchors and feature offsets. Similar to ODM in DRNet, RD takes over RG's outputs as references, and detect objects frame by frame. If RG only predicts refined anchors, the framework is called TRNet. When feature offsets are also predicted by RG, the anchor-offset detection with a deformable detection head is also employed by RD, and we call this structure TDRNet. That is, compared to TDRNet (see Fig. 5(a)), TRNet does not contain the module of feature location refinement, and its detection head is composed of traditional convolutions.

B. Training
In general, temporal detectors usually have a complex training process with sequential images. For example, TSSD developed a multi-step training strategy [16], and the initialization for multi-frame regression layer in TPN is complicated [17]. Conversely, the training process for TRNet and TDRNet is refreshingly concise, and we also eliminate the need of sequential training images. As shown in Fig. 5(b), during the training process, RG and RD play similar roles to DRNet's ARM and ODM, respectively. Thereby, both RG and RD can be trained with static images following DRNet's basic training settings and loss functions.

C. Inference
Then, RD detects objects with I m , ar, and ∆p, where f Im is the feature extracted from I m .
Despite the similar detection manner, it is apparent that RD is more computationally efficient than DRNet. Therefore, considering the temporal context in temporal vision, a key frame duration is used for RG to pursue a better tradeoff between accuracy and speed. That is, only key frames will be processed by RG, while non-key frames are detected by RD with previous RG's outputs. Mathematically, in (7), ar = ar m , ∆p = ∆p m for key frames, whereas ar, ∆p are from the previous key frame when detecting non-key frames. In this manner, ar and ∆p are propagated as the temporal information. As illustrated in Fig. 5(c), using the first frame in a period, RG generates refinement references that will survive k time stamps. Then, RD detects objects based on these references in the whole period. It is apparent that frequent Fig. 7. Visualization of refined feature sampling locations for Conv5 3. For better visualization, only the sampling centers (i.e., the center dot in left-bottom Fig 3(b)) are demonstrated. The original sampling centers are illustrated with green dots, which are regularly tiled on images. The red dots show the refined sampling centers that have a stronger capability of describing objects. These images are from VOC, COCO and ImageNet VID. reference update would lead to higher detection accuracy and more computational costs, so the trade-off between accuracy and speed can be adjusted by different k setting.
Taking aim at adapting to various object motion, a soft refinement strategy is proposed with a soft coefficient e. In SSD, the intent of designing anchor is to use numerous boxes to cover the whole image as the prior knowledge, but significantly discarding anchor diversity, the refined anchors tend to surround foreground. Refined anchors are in favor of static detection, but objects in videos have a variety of motion properties or pose changes. Hence, the soft refinement strategy is designed to retain the anchor diversity for relatively long temporal detection period. The soft refinement can be given as ar s = ar × e, where ar s is the soft anchor offset, and as a scalar, e ∈ [0, 1] multiplies each element in a tensor. Because ar is the offset from original anchors, ar × e can relax the intensity of anchor refinement by reducing offset magnitude so that refined anchors can be loosely scattered around objects. Referring to Fig. 6, refined anchors are computed by the first frame in the period, then some key refined anchors are visualized in the 2nd, 4th, and 8th frames. Without the soft refinement strategy, they gradually fail to be precisely aware of objects across time. For example, when e = 1, refined anchors cannot surround the head of a sheep in the 8th frame. This phenomenon causes regression difficulties for RD, prohibiting k from increasing. That is, e = 1 incurs that the refinement information can hardly be propagated in a relatively long range of time series. When e = 0.75 or 0.5, this drawback is mitigated so that the detection period can be longer for a better trade-off between accuracy and speed.  [19], COCO [20], and ImageNet VID [18] datasets. Furthermore, we applied TDRNet to online underwater object detection.

A. Ablation Studies of DRNet320-VGG16 on VOC 2007
Experiments on PASCAL VOC 2007 are first conducted with VGG16 [38] as the backbone to study the proposed dual refinement mechanism in detail. In this section, the models are trained on the union set of VOC 2007 trainval and VOC 2012 trainval (16, 551 images, denoted as "07+12"), and evaluated on VOC 2007 test set (4, 952 images). We use mAP to describe the detection accuracy. For the convenience of comparison, RefineDet without negative anchor filtering is employed as the baseline, whose mAP is 79.1% in our reproduced Pytorch implementation (Note that it is 79.5% in original Caffe implementation). The changes of mAP caused by various model designs are shown in Table I. 1) Anchor-Offset Detection: The anchor-offset detection contains an anchor refinement, a feature location refinement, and a deformable detection head, the first of which has been studied by [6], so we focus on the latter two components. At first, the deformable detection head without feature location refinement is tested. Following [35], the offsets are computed with ODM features (referring to (2)). As a result, this change leads to a 0.8% mAP drop. In our opinion, this should be attributed to improper offsets. That is, the refined anchors are computed with ARM while the feature offsets are from ODM, so they are independent, making refined feature locations still fail to describe refined anchor regions.
The refined anchors have been displayed in Fig. 6, so the refined feature sampling locations are also demonstrated in Fig. 7 to better explain the advantages of the proposed anchoroffset detection. For better visualization, only the sampling centers (i.e., the center dot in left-bottom Fig. 3(b)) are demonstrated. Referring to green dots in Fig. 7, the pre-defined detection features are regularly fixed on feature maps (their locations are mapped to the original images for visualization). This design is justified for the traditional SSD since anchors are also tiled in the same manner. However, the refined anchors tend to surround objects for more precision localization (see Fig. 6), so it is reasonable that the feature locations should have the same tendency. As shown with red dots, gathering towards objects, the refined feature locations are more suitable for regression and classification. Moreover, in some areas away from objects, the refined feature locations would not blindly shift towards targets so that the detection capability for the whole image can be maintained.
Therefore, the operation of the proposed feature location refinement is crucial to capture accurate detection features. Following the pipeline of anchor-offset detection, the refined feature locations are tightly associated with refined anchors. Thus, a 0.7% mAP rise (i.e., 79.8% vs. 79.1%) is induced.
2) Multi-Deformable Head: For leveraging more contextual information for detection, multiple detection paths are devised with various respective field sizes, or convolution kernel size and dilation. The effectiveness of various multi-deformable designs is shown in Table II. At first, the 1 × 1 grid is employed to utilize shrunken region-level features, but it incurs negligible effectiveness. The 1 × 1 grid should have focused on most suitable local parts for detection, but feature offsets are computed with anchor offsets in our pipeline, ignoring suitable local parts. Then, the 3 × 3 grid with dilation is devised as one of the detection paths, but it leads to a 0.4% drop in mAP. Although it expands the respective field, the dilated 3 × 3 grid splits features and fails to describe objects effectively. Covering this shortage, the 5 × 5 grid without dilation works more effectively, and it invites a 0.7% mAP rise (i.e., 80.5% vs. 79.8%) since more contextual information is involved. Moreover, the 1 × 1 detection path is removed, and this more efficient design still can reach 80.3% in mAP. These comparisons also indicate that the improvement comes from above-analyzed reasons rather than increasing parameter size.
3) Towards More Effective Training: Batch normalization [41] is introduced to the backbone for more effective training, and a significant improvement in accuracy is incurred, i.e., 81.1% mAP. Then, the anchor-offset detection and multideformable head further boost the performance. Referring to Table I, removing the multi-deformable head leads to a 0.3% drop in mAP, and removing the anchor-offset detection invites another 0.6% mAP drop. Thus, our designs are still efficient, making a superior detection performance with such a small input image, i.e., 82.0% mAP and 320 × 320 input size.

B. Results on VOC 2007
We use the initial learning rate of 0.001 for the first 130 training epochs, then use the learning rate of 0.0001 for the next 40 epochs and 0.00001 for another 40 epochs. Referring to Table III, our DRNet320 achieves 82.0% mAP surpassing all methods with such small inputs by a large margin. When compared to SSD300, our method outperforms it by 4.8 points (i.e., 82.0% vs. 77.2%), and DRNet320 further improves mAP by 2.0% as for RefineDet320 (i.e., 82.0% vs. 80.0%). Compared to RFBNet300, our DRNet320 also has 1.5-point higher mAP (i.e., 82.0% vs. 80.5%).
For 512 × 512 input size, DRNet512 obtains 82.8% mAP that is also competitive with all compared methods. Only Attention CoupleNet [22] has slightly higher mAP than ours (i.e., 82.8% vs. 83.1%). However, Attention CoupleNet uses ResNet101 [42] as its backbone, and its results come with larger input size (i.e., 1000 × 600). Besides, Attention Cou-pleNet introduces extra segmentation annotations to its multiscale training processing. In addition, DRNet512's inference speed surpasses that of Attention CoupleNet by a large margin (i.e., 32.2 vs. 6.9 FPS). Therefore, the proposed DRNet achieves a better trade-off between accuracy and speed. To relieve the impact of relatively small input size, we leverage multi-scale strategy for testing, and DRNet320 and DRNet512 can obtain 83.9%, 84.4% mAP, respectively.

C. Results on VOC 2012
More challenging VOC 2012 dataset is employed to evaluate our proposed designs, and we use the union set of VOC

D. Results on COCO
We perform a thorough analysis on COCO detection dataset, which contains 80 class labels. As in previous work, we also use the union of training images and a subset of validation images (118, 278 images, denoted as "trainval35k") for training, and test models on test-dev set (20,     As shown in Table IV, some anchor-free methods recently achieve high AP on COCO, which leverage key-point technology [30]- [32] or full convolution [33] for object detection. DRNet320 achieves the results of 30.5%, which is better than contemporary methods (e.g., RefineDet320, RFBNet300), so our approach can effectively cope with a variety of complex situations with small input resolution. Furthermore, DRNet512 obtains a more competitive AP of 34.3%. Because they have similar AP results, we draw readers' attention to a deep comparison among methods in boldface. (Note that YOLOv3 is not written in bold because its input size is 608.) At first, DRNet512 has huge improvements as opposed to Re-fineDet512 on all criteria, where our designs are proved to be successful. Moreover, our DRNet512 has the best VOC-like AP@0.5 (i.e., 57.1%) and AP S (i.e., 17.6%), so our method is more adept at small object detection owing to the proposed dual refinement mechanism. However, our results on AP@0.75 and AP L are not comparable with that of some methods. This is caused by two reasons: i) two-stage methods use larger input size; ii) ResNet101 or RFB block [27] provides larger effective receptive field for describing large objects [43]. Therefore, our methods are also tested with ResNet101 as the backbone. As a result, more competitive results are induced, i.e., DRNet320-ResNet101 delivers 33.5% AP and DRNet512-ResNet101 achieves 38.6% AP. If multi-scale testing is employed, we see 42.4% AP from DRNet512-ResNet101. Additionally, using MobileNet [44] as the backbone, our DRNet outperforms Faster RCNN, SSD, and RFBNet by a substantial margin. DRNet and RFBNet are similar in VOC mAP and COCO AP. Although similar performances are produced, DRNet and RFBNet are designed based on different motivations. That is, DRNet solves the problem of inaccurate anchors and feature locations, while RFBNet enhances the receptive field of the backbone. Two ideas are complementary so that anchor-offset detection and RFB block can be employed simultaneously.   Error analysis of DRNet512 is conducted on COCO 2014 minival set (5, 000 images), and precision-recall curves are shown on person, vehicle, furniture, and electronic classes. From Fig. 8, it is seen that there exists room for improvement of location precision. As for classification, DRNet has less confusion with similar categories or others (Sim & Oth). Thus, our approach is good at inter-class inference, benefiting from accurate single-stage detection features generated by the feature location refinement. By contrast, the error caused by the background (BG) is slightly serious. Probable improvement proposals will be discussed in Section V-G.

E. Results on ImageNet VID
TRNet and TDRNet are evaluated on ImageNet VID dataset, which requires algorithms to detect 30-class targets in consecutive frames. There are 4, 000 videos in the training set (1, 181, 113 frames), and 555 videos in the validation set (176, 126 frames). The initial learning rate is 0.001 for the first 70 epochs, then we use a learning rate of 0.0001 for the next 30 epochs and 0.00001 for another 30 epochs. For fast inference speed, all models use 320 × 320 input images.
1) Accuracy vs. Speed Trade-Off: SSD with 4-scale detection features serves as the baseline, called SSD4s, and RG and RD are also contrasted with similar structure (see Fig 5(a)). As a result, SSD4s-VGG16 and SSD4s-MobileNet obtain 63.0%, 58.3% in mAP, respectively. The key frame duration is used for temporal detection, so accuracy vs. speed tradeoff based on k is first analyzed. As shown in Fig. 9(a), TRNet significantly improves the mAP by 3.6% (i.e., 66.6% vs. 63.0). As k increasing, the mAP decreases while the speed raises. Note that NMS impacts detection speed to some extent, but this part is out of the scope of this paper, so the FPS without NMS is also reported (denoted as Forward FPS). As plotted in Fig. 9  To overcome TRNet's rapid mAP decrease with increasing k, the soft refinement strategy is introduced to TRNet with a soft coefficient e. Referring to Fig. 9(d), e = 0.5 can restrict this mAP drop within 1% from k = 1 to k = 8, i.e., 66.6% vs. 66.2% for VGG16 and 61.3% vs. 61.0% for MobileNet.
As shown in Fig. 10, with 320 × 320 input size, this paper presents a series of approaches for the trade-off between accuracy and speed. The fast solution is TRNet-MobileNet (k = 8), whose inference time is 14 ms. The most accurate method in this paper is DRNet320 with 69.4% mAP and 25 ms in inference. In terms of TDRNet-VGG16 and DRNet320-VGG16, it can be seen that DRNet is more accurate owing to FPN and multi-deformable head, yet TDRNet has a better trade-off between accuracy and speed.
2) Comparison with Other Architectures: TRNet and TDR-Net are compared against several prior and contemporary approaches in Table V. Existing video detectors are categorized into offline methods (i.e., batch-frame mode) and online methods. Most methods are based on a two-stage detector and a deep backbone, so they usually have high mAP yet impractical execution time. As for offline approaches, this non-causal batch-frame mode usually leverages both previous and future information that prohibits it from real-world applications. In addition, recent works usually borrow other temporal modules (e.g., tracker, optical flow, and LSTM) to integrate multiframe information. Among single-stage methods, TDRNet-VGG16 has a significant superiority in accuracy, i.e., 1.9% and 12.9% higher mAP than TSSD [16] and LSTM-SSD [15], respectively. When compared to MobileNet-based detectors, TDRNet-MobileNet has the best results, i.e., it outperforms LSTM-SSD by 8.7 points and surpasses HPVD-Mob [12] by 2.9 points. To the best of our knowledge, our designs have the following merits: i) instead of borrowing other temporal modules, temporal information is exploited from the detector itself. Thus, our design is a new online detection mode for videos; ii) TDRNet achieves the highest mAP among realtime online temporal detectors, and it induces a better trade-off between accuracy and speed for real-world tasks.

F. Underwater Object Detection and Grasping
Underwater missions are quite intractable for humans, so we use an autonomous system for these difficult tasks, i.e., underwater navigation and object grasping. Owing to the characteristic of good accuracy vs. speed trade-off, TDRNet is suited to real-world applications. In reality, we employ a remote operated vehicle (ROV) for underwater grasping, where a microcomputer with an Intel I5-6400 CPU, an NVIDIA GTX 1060 GPU, and 8 GB RAM is deployed and a camera is placed in the electric compartment for visual navigation. Based on our proposed method, the ROV is able to approach targets and grasp marine products (e.g., sea cucumbers, sea urchins, bivalves, and starfish) using a manipulator. The test venue is located in Zhangzidao, China, where the water depth is approximately 10 m. It should be noted that the dataset is from http://en.cnurpc.org. Based on this dataset, we compare some prior approaches and DRNet. Referring to Table VI, the proposed DRNet has advantages in accuracy for this difficult real-world mission. Note that Table VI does not include TRNet and TDRNet because this is a static dataset.
As shown in Fig. 11(a), the ROV works on a natural seabed, and TDRNet is competent in detecting objects in an unstructured undersea environment (see Fig. 11(b)). For better visualization, we demonstrate the detection results of sea cucumbers, sea urchins, and bivalves using yellow, cyan,  [11] for key frame selection, and we employ a pre-fixed key frame duration. We argue that the adaptive key frame scheduling is needless for both accuracy and speed in this paper: i) any scheduling strategy cannot generate an mAP that outmatches the result of k = 1. Therefore, given that the speed of RG is fast enough (i.e., 270 FPS) and a scheduling strategy should deal with each frame, it is better to set k = 1 than to use an adaptive key frame scheduling. ii) the longest period is k = 8 in our experiments, and we also state that longer detection period is needless: Forward FPS of SSD4s-VGG16 is 270 and that of TRNet-VGG16 reaches 234 (k = 8). Thus, longer key frame duration has an ignorable contribution to inference speed since TRNet and TDRNet cannot overpass SSD4s in Forward FPS.
2) Further Enhancement of Refinement Networks: In terms of accuracy, it can be seen from Fig. 8 that there still exists room for improvement of location precision and foregroundbackground classification. We present two probable solutions: i) multi-step refinement could be beneficial; ii) because of the hard negative mining, only a part of negative samples (i.e., background) are used for training. Therefore, using a focal loss [24] to train a network with all negative samples could be more effective. For example, Chi et al. used focal loss and negative anchor filtering to train a refinement network and achieved high performance on face detection [45].
Regarding inference speed, the NMS has an impact. There could exist two solutions: i) decreasing the anchor amount could be beneficial; ii) an end-to-end detector is becoming urgently necessary. For example, Hu et al. developed a relation network for both detection and duplicate removal, so the whole network can perform in an end-to-end manner [46].
3) Challenging Object Detection in Real-World Scenes: Real-world object detection is more challenging, but classical datasets, e.g., VOC, COCO, and VID, could not contain various real-world environments. Thus, it important to study object detection in specific real-world scenes. To this end, many realworld datasets are developed. For example, besides our studied underwater dataset, TU-VDN dataset [47] introduces adverse weather conditions to object detection.

VI. CONCLUSION AND FUTURE WORK
In this paper, we have taken aim at precisely detecting objects in real time for static and temporal scenes. Firstly, drawbacks of the single-stage detector are analyzed from the strengths of two-stage methods. Thereby, including an anchor refinement, a feature location refinement, and a deformable detection head, a novel anchor-offset detection is proposed. Besides two-step regression, the anchor-offset detection is also able to capture accurate single-stage features for detection. Correspondingly, a DRNet is proposed based on the anchoroffset detection, where a multi-deformable head is also designed for more contextual information. In the case of temporal detection, we propagate the refinement information in the anchor-offset detection across time and propose a TRNet and a TDRNet with a reference generator and a refinement detector. Our developed approaches have been evaluated on PASCAL VOC, COCO, and ImageNet VID. As a result, our designs induce a considerably enhanced detection accuracy and see a substantial improvement on the trade-off between accuracy and speed. Finally, the proposed algorithms are applied to underwater object detection and grasping.
In the future, we plan to incorporate attention mechanism to the anchor-offset detection and design more effective networks for more robust feature learning.