{'fraction_non_alphanumeric': 0.03868828297715549, 'fraction_numerical': 0.017399492344223368, 'mean_word_length': 4.79113323850166, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adaptive Boundary Proposal Network for Arbitrary Shape Text Detection

Arbitrary shape text detection is a challenging task due to the high complexity and variety of scene texts. In this work, we propose a novel adaptive boundary proposal network for arbitrary shape text detection, which can learn to directly produce accurate boundary for arbitrary shape text without any post-processing. Our method mainly consists of a boundary proposal model and an innovative adaptive boundary deformation model. The boundary proposal model constructed by multi-layer dilated convolutions is adopted to produce prior information (including classification map, distance field, and direction field) and coarse boundary proposals. The adaptive boundary deformation model is an encoder-decoder network, in which the encoder mainly consists of a Graph Convolutional Network (GCN) and a Recurrent Neural Network (RNN). It aims to perform boundary deformation in an iterative way for obtaining text instance shape guided by prior information from the boundary proposal model. In this way, our method can directly and efficiently generate accurate text boundaries without complex post-processing. Extensive experiments on publicly available datasets demonstrate the state-of-the-art performance of our method.


Introduction
Scene text detection has been widely applied in various applications, such as online education, product search, and video scene parsing. Benefiting from the rapid development of deep learning, text detection methods [29,11,42,22] have achieved impressive performance on images in which text instances are regular shape or aspect ratio. Recently, arbitrary shape text detection has attracted ever-increasing interests for it can well adapt to real applications.
Although arbitrary shape text detection methods [2, 12, * Corresponding author.  41] have achieved great improvements in recent years, there are still many issues to be addressed due to the challenging characteristic of scene texts, including varied shape, texture, scale, etc. Segmentation-based methods [37,33,12] have sparked a new wave of arbitrary shape text detection that locate text regions by a pixel-level prediction for enhancing the robustness to shape variations. However, there are still two main problems that remain to be explored.
One problem is that segmentation-based methods tend to fail in separating adjacent text instances in image. To solve this problem, existing methods [19,34,30,40] usually shrink annotation boundaries as kernels (e.g., text kernel [34], text center region [19]) to distinguish different text instances. For rebuilding a complete text, these methods usually need to merge the pixels in text regions to kernels by pre-defined expansion rules or auxiliary information (e.g., similarity vector [34]). However, the merging process in [37,33] are always performed by pixel-to-pixel, which is complex and inefficient. The other problem is that the final detected contours of texts in existing segmentation-based methods usually contain a lot of defects and noises. This because the performance of existing segmentation-based methods [4,12,33,34] greatly relies on the accuracy of contour detection, neglecting adaptively adjusting detected contours. Different from generic object instances, text instances usually don't have closed contours and often contain a lot of background noisy pixels in coarse-grained boundary annotations. These will generate unpredictable results of pixels, especially those near boundaries, resulting in a lot of noises and defects in segmentation results.
To tackle the above-mentioned problems, we propose a novel adaptive boundary proposal network for arbitrary shape text detection, which can learn to directly produce accurate boundary for arbitrary shape text without any postprocessing. Our adaptive boundary proposal network is mainly composed of a boundary proposal model and an adaptive boundary deformation model. The boundary proposal model is composed of multi-layer dilated convolutions, which will predict a classification map, a distance field, and a direction field based on shared convolutions. Inspired by RPN [25], we adopt the distance field and pixel classification map to generate coarse boundary proposals as shown in Fig. 1 (a). These coarse boundary proposals can roughly locate texts, and well separate adjacent texts because they are always slimmer than their boundary annotations in our method. To refine the coarse proposals, we adopt an innovative adaptive boundary deformation model to perform iterative boundary deformation for generating accurate text instance shape under the guidance of prior information (classification map, distance field and direction field). For fully excavating and exploiting topology and sequence context in each boundary proposal, the adaptive boundary deformation model adopt an encoderdecoder structure, in which the encoder mainly consists of a GCN and a RNN (B-LSTM). Notably, the proposed method is a unified end-to-end trainable framework with iterative optimization. Extensive experiments demonstrate that our method achieves state-of-the-art performance on several publicly available datasets.
In summary, the main contributions of this paper are three-fold: • We propose a novel unified end-to-end trainable framework for arbitrary shape text detection, which can directly generate accurate boundaries of arbitrary shape texts without any post-processing. • We propose an adaptive boundary deformation model which can perform iterative boundary deformation for refining text boundary. • Extensive experiments on public available datasets demonstrate the state-of-the-art performance of our method.

Related Work
Regression-Based Methods. Methods of this type rely on box-regression based object detection frameworks with word-level and line-level prior knowledge [22,10,13,42,40,8]. Different from generic objects, texts are often presented in irregular shapes with various aspect ratios. To deal with this problem, RRPN [22] and Textboxes++ [10] localized text boxes by predicting the offsets from anchors. Different from these methods localizing text regions by implementing refinement on pre-defined anchors, EAST [42] and DDR [7] directly regressed the offsets from boundaries or vertexes to the current point for accurate and efficient text detection. Although regression-based methods have achieved good performance in quadrilateral text detection, they often can't well adapt to arbitrary shape text detection.
CC-Based Methods. The Connected Component (CC) based methods [39,26,5,2,41] usually detect individual text parts or characters firstly, followed by a link or group post-processing procedure for generating final texts. CRAFT [2] detected text regions by exploring affinities between characters. TextDragon [5] first detected the local region of the text, and then groups them by their geometric relations. Zhang et al. [41] used a graph convolution neural network (GCN) to learn and infer the linkage relationships of text components to group text components. Although CC-based methods have a more flexible representation that can adapt to irregular shape text well, the complex postprocessing for grouping text components to complete texts is always time-consuming and unsatisfied.
Segmentation-Based Methods. Methods of this type [33,37,34,12,20] mainly draw inspiration from semantic segmentation methods and detect texts by estimating word bounding areas. To effectively distinguish adjacent text, PSENet [33] adopted a progressive scale algorithm to gradually expand the pre-defined kernels. In [34], Wang et al.
proposed an efficient arbitrary shape text detector, named Pixel Aggregation Network (PAN), which is equipped with a low computational-cost segmentation head and learnable post-processing. DB [12] performed an adaptive binarization process in a segmentation network, which simplifies the post-processing and enhances the detection performance. However, the performances of these methods are strongly affected by the quality of segmentation accuracy.
Contour-based methods. In addition to the above methods, Contour-based methods also have attracted many researchers [21,35,32,36]. Yao et al. [21] detected texts by predicting the corner of texts, and Lyu et al. [35] adopted a similar architecture as SSD [16] and rebuilt text with predicted corner points. Wang et al. [32] proposed an end-to-end approach toward arbitrary shape text spotting, which proposed a boundary point detection network to locate the text boundary in the text proposal region. Contour-Net [36] represented text region with a set of contour points, which adopted a Local Orthogonal Texture-aware Module (LOTM) to model the local texture information of proposal features in two orthogonal directions for generating contour points. However, compared with segmentation-based methods, the contour-based method has a big gap in detection performance and speed without recognition information.

Overview
The framework of our method is illustrated in Fig. 2. The ResNet-50 [6] is adopted to extract features. To preserve spatial resolution and take full advantage of multi-level information, we exploit a multi-level feature fusion strategy (similar to FPN [14]), as shown in Fig. 3. The boundary proposal model composed of multi-layer dilated convolutions uses the shared features for performing text pixels classification, generating the distance field and direction field [37]. Then, we use these information to produce coarse boundary proposals. Each boundary proposal consists of N points, representing a possible text instance. For refining the coarse proposals, an adaptive boundary deformation model is proposed to perform iterative boundary deformation for obtaining the more accurate text boundaries under the guidance of prior information (classification map, distance field, and direction field).

Boundary Proposal Generation
The boundary proposal model is composed of multi-layer dilated convolutions, include two 3 × 3 convolution layers with different dilation rates and one 1×1 convolution layers, as shown in Fig. 2. It will use the shared features extracted from the backbone network to generate classification map, distance field map, and direction field map.
Similar to other text detection methods [42,19,41], the classification map contains a classification confidence of each pixel (text/non-text). As in [37,31], the direction field map (V ) consists of a two-dimensional unit vector ( → x , → y ), which indicates the direction of each text pixel in boundary to its nearest pixel on boundary (pixel-to-boundary). For each pixel p inside a text instance T , we will find its near- est pixel B p on text boundary, as shown in Fig. 5. Then, a two-dimensional unit vector V gt (p) that points away from the text pixel p to B p can be formulated as where − − → B p p represents the distance between B p and text pixel p, and T represent the total set of text instances in an image. For the non-text area (p ∈ T), we represent those pixels with (0, 0). The unit vector V gt (p) not only directly encodes approximately relative location of p inside T and highlights the boundary between adjacent text instances [37], but also provides direction indication infor-  For boundary deformation, the relative position distance information is as important as the direction information. In this work, the distance field map (D) is a normalized distance map, and the normalized distance of the text pixel p to nearest pixel B p on text boundary is defined as For the non-text area (p ∈ T), we represent the distance of those pixels with 0. L represents the scale of text instance T where the pixel p is located, and is defined as D gt (p) not only directly encodes the relative distance of p inside T and further highlights the boundary between ad-jacent text instances, but also provides a relative distance information for boundary deformation.
With the distance field map (D), we can generate candidate boundary proposals by using a fixed threshold (th d ) to the predicted distance, as shown in Fig. 4 (b). However, these candidate boundary proposals inevitably contains false detections. Hence, we calculate the average confidence of each candidate boundary proposal according to the classification map for removing some of them with low confidence (th s ), as shown in Fig. 4 (c) and (d).

Adaptive Boundary Deformation
Inspired by interactive annotation of object segmentation methods [1,15] and instance segmentation method [23], we perform arbitrary shape text detection by deforming a boundary proposal to a more accurate text boundary. Specifically, we take a boundary as input based on CNN features and predict per-vertex offsets pointing to the text boundary. In [1], the authors use a Recurrent Neural Network to model the sequence of 2D vertices of the polygon outlining an object. Afterwards, they propose a method [15] to treat the location of each control point as a continuous random variable, and learn to predict these variables via a Graph Neural Network that takes image evidence as input. Inspired by [1,15], DeepSnake [23] performs object segmentation by deforming an initial contour to object boundary with circular convolution which consists of multi-layer 1-D convolutions with 1 × N kernel size. However, these methods only consider individual topology context or sequence context, which isn't highly satisfactory for text detection. For the unique properties of texts, both topology context and sequence context are important in detection.
For each boundary proposal, we will uniformly sample N control points for facilitating batch processing. As shown in Fig. 2, the sampled control points form a closed poly-gon in which topology context and sequence context are co-existing. To fully exploit the boundary topology and sequence context, we introduce an adaptive boundary deformation model combined with GCN and RNN, which can efficiently perform feature learning and iterative boundary deformation for refining coarse text boundaries.
Let cp i = [x i , y i ] T denote the location of the i-th control point, and p = {p 0 , p 1 , ..., p N −1 } be the set of all control points. For a boundary proposal with N control points, we first construct feature vectors for each control point. The input feature f i for a control point cp i is a concatenation of 32-D shared features F s obtained by CNN backbone and 4-D prior features F p (e.g., pixels classification, distance field and direction field). Therefore, the features of a control point are extracted from the corresponding location in F : After obtaining the feature matrix X (size: N × C) of boundary proposal, we adopt the adaptive deformation model based on an encoder-decoder architecture to perform efficient feature learning and iterative boundary deformation. The encoder model is combined with GCN and RNN for feature learning, which can fully exploit and fuse the boundary topology and sequence context. As shown in Fig. 2, the encoder model can be formulated as where "⊕" refers to the concatenation operation; RNN is consisted of one-layer B-LTSM with 128 hidden size; Conv 1×1 consists of one-layer 1 × 1 convolution layers with 128 dimensions, which form a residual connection like RestNet [6]; GCN is consisted of four graph convolution layers activated by ReLU, and the graph convolution layer in our method is formulated as where X ∈ N ×di , X g ∈ N ×do , d i /d o is the dimension of in/out features, and N is the number of control points; G is a symmetric normalized Laplacian of size N × N ; W is a layer-specific trainable weight matrix;Ã = A + I N is an adjacency matrix of the local graph with added selfconnections; I N is the identity matrix andD is a diagonal matrix withD ii = jÃ ij . We form A by connecting each control point in p with its four neighbors. The decoder in adaptive deformation model consists of three-layer 1 × 1 convolutions with ReLU, which will learn to predict offsets between control points and the target points. To obtain the more accurate text boundary, we perform iterative boundary deformation, as shown in Fig. 2.

Optimization
In this work, the total loss L can be formulated as where L Bp is a loss for the boundary proposal model, and L Bd is a loss for the adaptive boundary deformation model; eps denotes the maximum epoch of training, and i denote the i-th epoch in train. In our experiments, λ is set to 0.1. In Eq. 7, L Bp is computed as where L cls is a cross-entropy classification loss for pixels classification, and L D is a L 2 regression loss for distance field. OHEM [27] is adopted for L cls and L D in which the ratio between the negatives and positives is set to 3:1.
To balance the losses in L Bp , the weights α is set to 3.0. Similar to [31], L V consists of L 2 -norm distance and angle distance for direction field V: where Ω represents image domain; the weight (w(p) = 1/ |GT p |) at pixel p is proportional to the inverse square root of the size of ground truth segment GT p containing p. L Bd is a point matching loss similar to [15]. In this work, the prediction and ground truth control point sets have equal sized and similar order (counter-clockwise), denoted as p = {p 0 , p 1 , · · · , p N −1 }, and p = {p 0 , p 1 , · · · , p N −1 } (N is the number of points), respectively. Hence, the matching loss for p and p is defined as Because there is usually more than one text instance in an image, L Bd is defined as where T represents all the text instances in an image, p represents the control point set for text instance T (T ∈ T).

Datasets
Total-Text: It consists of 1, 255 training and 300 testing complex images, including horizontal, multi-oriented, and curved text with polygon and word-level annotations. CTW-1500: It consists of 1, 000 training and 500 testing images, and curved text instances are annotated by polygons with 14 vertices. MSRA-TD500: It consists of 500 training and 200 testing images, including English and Chinese texts which contain multi-lingual long texts with multi-orientations. SynthText: It contains 800k synthetic images generated by blending natural images with artificial text which are all word-level annotated . ICDAR2017-MLT: It consists of 7, 200 training images, 1, 800 validation images, and 9, 000 test images with multilingual (9 languages) texts annotated by quadrangle.

Implementation Details
In our experiments, we first pre-train our network on SynthText by one epochs, in which images are randomly cropped and resized to 512 × 512. In pre-training, the Adam [9] optimizer is applied with a fixed learning rate of 0.001, and a mini-batch is set to 16. In fine-tuning, we randomly crop the text region, and resize them to 640 × 640 for training the model with the mini-batch 12. The Adam [9] optimizer is adopted, in which the initial learning rate is 0.001 and decreased to 0.9 of the original after each 50 epochs. The data augmentation includes: random rotation with an angle (sampled by Gaussian distribution in (−60 • , 60 • )), random cropping, and random flipping. In inference, we keep the aspect ratio of test images, then resize and pad them into the same size for testing. The code is implemented with PyTorch 1.7 and python 3. Training is performed on single GPU (RTX-3090), and testing is performed on single GPU (GeForce RTX-2080) with Intel Xeon Silver 4108 CPU @ 1.80GHz.

Ablation Study
In ablation experiments, we only train the model on corresponding real-world datasets for 660 epochs without pretraining, and the other training settings are identical with the fine-tuning process in Sec 4.2. In testing, the short side of an image is scaled to 640, and ensure that the long side doesn't exceed 1,024. The threshold th d and th s are set to 0.3 and 0.8, respectively. Effectiveness of adaptive deformation model. To verify the effectiveness of the adaptive deformation model, we conduct ablation experiments on Total-Text and CTW-1500. Our deformation model consists of an encoder and a decoder. For fair comparison, we use a lightweight full connection network (FC) structure as decoder, and we adopt four types of encoder, i.e., FC with Conv 1×1 , RNN, circular convolution, and graph convolution (GCN), for conducting comparative experiments. As listed in Tab. 1, our adaptive deformation model achieves the best performance compared with the other four methods on both Total-Text and CTW-1500, which achieves improvements by 0.94% in terms of F-measure on Total-Text compared with RNN, and by 0.85% in terms of F-measure on Total-Text compared with GCN. Moreover, our adaptive deformation model doesn't bring obvious more consume of detection time. Influence of control point number (N ). We study the influence of the control point number (N ) by setting the number of control point number from 12 to 32 with an interval of 4. The models are evaluated on Total-Text and CTW-1500. From Fig. 6, we can find that the F-measures drops when N is too large or too small. Especially, too few control points will make great degradation of performance. This is because the detection boundary often can't correctly cover the whole text when the control number is too small. Apparently, our model achieves the best detection performance on both two datasets while the number of control points is around 20. Therefore, in our experiments, the number of control points is fixed at 20. Influence of iteration number. To fully validate the influence of iteration number, we further compare models with different inference iterations. As listed in Tab. 2, with the increase of the number of iterations, the detection performance is gradually improved but the inference speed of our model is gradually dropped. When the number of iterations is from 2 to 3, the increase of detection performance is not very obvious. Considering the balance of speed and performance, the number of iterations is set to 3 by default in our experiments. As shown in Fig. 7, the detection boundaries become more accurate along with the increase of iterations.  Influence of prior information. We conduct ablation studies on Total-Text to verify the importance of each prior information (e.g., classification map, distance field and direction field). As listed in Tab. 3, the detection performance is unsatisfactory when only use classification map. Once distance field and direction field are introduced, the performance is improved significantly, and F-measure is improved by 5.45% and 1.55%, respectively.

Comparison with the state-of-the-arts
We compare our method with the previous state-of-theart methods on Total-Text, CTW-1500 and MSRA-TD500. In testing, the short side of the images is scaled to 640, and  Fig. 8 (a) and Fig. 8 (b), we can observe that our method can precisely detect word-level irregular texts.  CTW-1500. In testing, the threshold th s is set to 0.8. Representative visible results are shown in Fig. 8 (c) and (d), which indicate our method precisely detects boundaries of long curved text with line-level. The quantitative results are listed in Tab. 6. Compared with the previous sate-of-the-art methods [12,34,36], our approach achieves promising performance of both precision (87.81%) and Fmeasure (85.0%). Specifically, our method greatly outperforms TextSnake [19] and DB [12] on CTW-1500 in item of F-measure by 9.4% and 1.6%, respectively. MSRA-TD500. In testing, the threshold th s is set to 0.925. The quantitative comparisons with other methods on this dataset is listed in Tab. 7. From Tab. 7, we can conclude that our method successfully detects long text lines of arbitrary orientations and sizes. Notably, our method achieves 85.57% in terms of F-measure, which outperforms other sate-of-the-art methods, such as DB [12], DRRG [41], etc.

Conclusion
In this paper, we propose a novel adaptive boundary proposal network for arbitrary shape text detection, which adopt an boundary proposal model to generate coarse boundary proposals, and then adopt an adaptive boundary deformation model combined with GCN and RNN to perform iterative boundary deformation to obtain the more accurate text instance shape. Extensive experiments show that the proposed method can precisely detects the boundaries of arbitrary shape text in challenging datasets. In future study, we are interested in developing a real time method for text of arbitrary shapes based on current work.