{'fraction_non_alphanumeric': 0.028666693774651322, 'fraction_numerical': 0.021062904078396293, 'mean_word_length': 4.757022471910112, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Layer Aggregation

Convolutional networks have had great success in image classification and other areas of computer vision. Recent efforts have designed deeper or wider networks to improve performance; as convolutional blocks are usually stacked together, blocks at different depths represent information at different scales. Recent models have explored `skip' connections to aggregate information across layers, but heretofore such skip connections have themselves been `shallow', projecting to a single fusion node. In this paper, we investigate new deep-across-layer architectures to aggregate the information from multiple layers. We propose novel iterative and hierarchical structures for deep layer aggregation. The former can produce deep high resolution representations from a network whose final layers have low resolution, while the latter can effectively combine scale information from all blocks. Results show that the our proposed architectures can make use of network parameters and features more efficiently without dictating convolution module structure. We also show transfer of the learned networks to semantic segmentation tasks and achieve better results than alternative networks with baseline training settings.


Introduction
Representation learning has received considerable attention in computer vision community and great progress has been made on major challenges (Russakovsky et al. 2015). Learned representations on one task can be transfered to other computer vision problems, which further encourages research in this area (Krizhevsky et al. 2012;Szegedy et al. 2015;Simonyan and Zisserman 2015;He et al. 2016;Zagoruyko and Komodakis 2016).
Deeper networks with bigger receptive fields typically yield better performance. But deeper networks are usually more difficult to optimize. To overcome this barrier, existing works have proposed different building blocks or modules that can reduce the number of parameters while maintaining effective convolutional kernel size, e.g., in the inception module with dimensionality reduction (Szegedy et al. 2015), or the propagation of meaningful gradients through hundreds of layers in the residual block architecture proposed by (He et al. 2016).
Nevertheless, scant attention has been paid to organization of those blocks or modules other than sequential layering. LeCun et al. (1998) stacks several convolutional layers followed by pooling or non-linear activation together to extract features for digit recognition. Krizhevsky et al. (2012) extended this idea to make breakthrough progress in large-scale image recognition. Since then, even though more modules based on convolutional layers have been proposed, they are still generally stacked together to construct the final networks. Various network visualization methods (Zeiler and Fergus 2014; Yu et al. 2017) show that deeper layers in this stacking structure can extract semantic features at larger scale. This may be good for representing object-centric images, but it may not suffice to represent natural images with multiple objects in which objects can appear at different scale. Although previous works usually assume that the network can learn scale-invariant representation, it may not be the case in practice, especially when the amount of training data is relatively small. In this paper, we investigate how to aggregate multiple layer activations so that the final representation can incorporate scale information explicitly and the resulting network can be more compact. In contrast to the related works cited above, our layer aggregation connections can themselves be deep, being aggregated by learned intermediate units on the path to a final integration node. Instead of building new networks from scratch, we design a network framework in which the modules proposed in the previous works can be directly used and it can be compatible with future module design. Also, the network should be suitable for different computer vision problems and the learned representation can be transfered to different domains.
We propose two specific structures for deep layer aggregation: Iterative Deep Aggregation (IDA) and Hierarchical Deep Aggregation (HDA). IDA can produce deep high resolution representations from a network whose final layers have low resolution, while HDA can effectively combine layer information from all blocks. In HDA, the building modules are organized in a hierarchical tree structure and multi-layer information is aggregated by different levels of tree nodes. It is deep in the sense that the tree has multiple levels from the root of the hierarchy to the shallowest layer. The framework is general so that we can use different building modules in the same structure and better module can lead to better performance. In HDA, the aggregation depth of blocks grows with the log of the number of blocks. HDA is useful when the blocks have very similar scale information, while IDA is useful to aggregate subnetworks whose outputs have dramatic scale change.
In our experiments, we utilize residual blocks from ResNet (He et al. 2016) and ResNeXt (Xie et al. 2017) and conduct experiments on large-scale image classification, fine-grained classification and semantic image segmentation. Results show that our framework can achieve similar results as baseline ResNet or ResNext architectures with fewer parameters thanks to the layer aggregation, which enables more efficient and compact models. Our model achieves new state-of-the-art results for compact models. Without modification, our networks also obtain new state-of-the-art results on several fine-grained classification benchmarks. We further transfer the learned network to semantic image segmentation for CamVid (Brostow et al. 2009) and Cityscapes (Cordts et al. 2016), and beat the previous best method by 4 points with fewer parameters on CamVid dataset.

Deep Layer Aggregation
Our basic convolutional block is a small convolutional network. A block can be as simple as a convolutional layer followed by non-linear activation, as in AlexNet (Krizhevsky et al. 2012) and VGGNet (Simonyan and Zisserman 2015). It usually contains several layers and intra-block skip connections to facilitate gradient back propagation, such as the inception module in GoogLeNet (Szegedy et al. 2015) and the residual block in ResNet (He et al. 2016). Most existing networks can be viewed as stacking of convolutional blocks, as shown in Figure 1(a). In this scheme, each block enlarges the receptive field; evidence has shown that more blocks capture semantically meaningful information at larger scale. In a conventional stacking structure, the output is generated by the last convolutional block, which doesn't have flexible scale information.
To be exact, we define aggregation as a network layer that learns to combine outputs of different layers. We call a group of aggregations deep if the output of lowest aggregated layer passes through multiple aggregations.

Iterative Deep Aggregation
We propose to aggregate the information at different layers across the network directly. Although this is rarely considered for image classification, people have proposed combining deep low resolution feature maps and shallow high resolution maps directly to produce high resolution prediction. Usually, the stack of convolutional blocks are divided into sub-networks by certain criterion such as subnetwork output resolution. Then additional layers are added to combine layer outputs from larger to smaller scales. Prior work on 'shallow-skip' layers, e.g., U-Net (Ronneberger et al. 2015), FCN (Long et al. 2015) and FPN , provide connections that bypass traditional sequential layers. This concept is shown in Figure 1(b). However, in this scheme, the lowest aggregated layer only passes through one aggregation node, which leads to shallow aggregation for smallest scale.
We propose iterative deep aggregation shown in Figure 1(c). The aggregation starts from the smallest scale and the information is iteratively combined with bigger scale.
The iterative deep aggregation function I for a serious of inputs x 1 , ..., x n with ascendantly deeper semantic information can be formally defined as where N is the aggregation node.

Hierarchical Deep Aggregation
While iterative deep aggregation can combine outputs of sub-networks, it is less effective to combine individual blocks.
We propose another aggregation method that hierarchically combines the convolutional blocks as in Figure 1(d). In this structure, all the convolutional blocks get deep aggregation.
We can change the connection within the hierarchy to obtain deeper aggregation. Instead of sending the output of one block to another directly, we use the output of the substree as the input of block in the next substree, as shown in Figure 1(e). Then the inputs of the convolutional blocks are the aggregation of all previous blocks, which is richer than output of a single convolutional block.
Furthermore, we merge the parent aggregation node with its left child node, as shown in Figure 1(f). This can save the network parameters and the resulting aggregations are still deep based on our definition.
Our proposed hierarchical deep aggregation is formulated in Equation 2 below. T n is the function for a hierarchy with depth n.
where N is the aggregation node. R and L are defined as where B represents the convolutional block.

Model elements
Aggregation Nodes The main function of an aggregation node is to combine and compress the information from their inputs. The nodes learn to select important information from their input layers and then project the representation to a lower dimension so that its output has the same dimension with the input from the last convolutional block before the node in the network. The number of inputs for IDA is always two in this paper, while HDA nodes have variable number of inputs depending on hierarchy depth.
Although any convolutional block can be used as an aggregation node for combination and compression, we choose to use a single convolutional layer as shown in Figure 2(b). This approach can avoid adding too much overhead to build the aggregation structures and it is easier to back propagate gradients through the single layer node during training. In image classification networks, all the nodes use 1×1 convolution. In semantic segmentation, we add additional iterative deep aggregation to upsample the dilated feature maps. 3×3 convolution is used in this case.
Residual connections are important to construct very deep convolutional networks (He et al. 2016). We can also add residual connections as shown in Figure 2(c). However, it is not clearly useful in the aggregation node. In HDA, the shortest path from any block to the root is at most the depth of the hierarchy, so diminishing or exploding gradients may not appear along the aggregation paths. In our experiments, we find that residual connection in node can help HDA when the deepest hierarchy has 4 levels or more, while it may hurt for networks with smaller hierarchy. Our base aggregation, i.e. N in Equation 1 and 2, is defined by: where σ is the non-linear activation, and w i and b are the weights in the convolution. If residual connections are added, the equation becomes   (c) show two general designs of nodes with 3 inputs. In (b), the inputs are concatenated and passed to a convolutional layer. In (c), after batch normalization, an additional residual connection adds input from the last block to the node output to facilitate gradient propagation for networks with more than 100 layers.

Convolutional Blocks
Deep layer aggregation is a general architecture in the sense that it can be compatible with different kinds of convolutional blocks. It has no requirement on the internal block structure. In this paper, we study three kinds of convolutional blocks with residual connections. We reduce the ratio between output dimension and middle by half for both bottleneck blocks in He et al. (2016) and split block in Xie et al. (2017). In the split block, cardinality is 32. The blocks and their configurations are show in Figure 3.

Image Classification and Segmentation
We now discuss how to use the architectures and components proposed in the previous sections to build networks for image understanding tasks. To study the performance of our aggregated learned representation, we focus on image classification and semantic image segmentation, which only requires a single projection layer on top of the learned representation to make predictions. n, 3 x 3, n n, 3 x 3, n n/2, 1 x 1, n n, 1 x 1, n/2 n/2, 3 x 3, n/2 n/32, 1 x 1, n n/32, 1 x 1, n n/32, 3 x 3, n/32 n/32, 3 x 3, n/32

Classification Networks
The classification networks can be divided into several levels and the layers with in each level have the same spatial resolution. We follow the structure in ResNet, where the final feature maps downsample the input images by 32 and a global average pooling is applied before softmax classification. As each downsampling shrinks the previous resolution by 2, there are 6 levels in total with the first level operating on the original image resolution. Each level is treated as a sub-network in this paper. We use hierarchical deep aggregation within each sub-network to connect the convolutional blocks and use iterative deep aggregation to connect the sub-networks. We can easily combine these two aggregations by sharing the aggregation nodes. In this case, we only need to change the root node at each hierarchy by combining Equation 1 and 2 to where x is the output of the previous sub-network and S downsamples the spatial resolution of the input by 2. We use max pooling with kernel size 2 and stride 2 for downsampling. As in the DRN proposed by Yu et al. (2017), we used convolutional layers instead of max pooling in the first two levels. In the first level, a 7×7 convolution followed by a basic block (Figure 3(a)) is applied on the input image. The second level only has a basic block. For the other levels, we use the combination of HDA and IDA.
For a direct comparison of layers and parameters in different networks, we build networks with similar number of layers as ResNet-34, ResNet-50 and ResNet-101, although the exact same depth can not be achieved because of our new hierarchical structure. To test how efficient DLA can make use of parameters, we also build networks with much fewer parameters by shrinking the numbers of channels in the levels. The schemes of our tested networks are listed in Table 1.

Segmentation Networks
Semantic image segmentation can exploit the aggregation of multi-scale information since segmented objects can have dramatically different scales due to perspective distortion. We can easily adopt DLA for segmentation. Given a pretrained DLA, we can transform final feature map to higher resolution suitable for segmentation using dilated convolution (Yu and Koltun 2016). After dilation, the final feature maps downsample the original images by 8.
Further, we can use IDA to learn the upsampling of final feature map. We first project the outputs of Level 3 to Level 6 to 32 channels and then upsample the levels to the same resolution as Level 2. All the projection and upsampling parameters are learned together with the other network parameters. But upsampling is initialized by bilinear interpolation. Finally, we iteratively aggregate the features maps from Level 2 to Level 6 to learn a deep fusion of low and high level features. The aggregation nodes use 3×3 convolutions instead of 1×1. This is similar to hypercolumn classifier proposed by Hariharan et al. (2015). But hypercolumn classifier can be treated as shallow aggregation. Our aggregation also uses 3×3 convolution to combine pixel predictions in the neighbors. It is worth noting that we use two IDA in this case: one for connecting levels in the original classification network and the other for upsamping the predicted feature maps by 4 times.

Experiments
We test Deep Layer Aggregation networks on two tasks: image classification and semantic image segmentation. Our hypothesis is that for classification tasks, layer aggregation can combine informa-   Figure 3 and n is the channel number. Level 1 and 2 only show channel numbers. Level 3 to 4 show d-n and d is the depth of HDA in that level.
tion of different scales directly in general, which can make more efficient use of parameters and layer outputs than stacking of convolutional blocks. For fine grained classification, the DLA networks may learn better representation for detailed object difference through deep aggregation. On semantic segmentation, IDA can learn to upsample the pixel-level predictions, which may improve segmenting small objects with little additional parameters.

ImageNet Classification
We first train our networks on the ImageNet 2012 training set (Russakovsky et al. 2015). Similar to ResNet (He et al. 2016), training is performed by SGD for 120 epochs with momentum 0.9, weight decay 10 −4 and batch size 256. We start the training with learning rate 0.1, which is reduced by 10 every 30 epochs. We use scale and aspect ratio augmentation (Xie et al. 2017), but not color perturbation. For fair comparison, we also train the ResNet models with the same training procedure. It leads to slightly better results than those reported in the original paper.
We evaluate the performance of trained models on the ImageNet 2012 validation set. The images are resized so that the shorter side has 256 pixels. Then central 224×224 crops are extracted from the images and fed into networks to measure prediction accuracy.
DLA vs ResNet compares DLA networks to ResNets with similar number of layers and same convolutional blocks as shown in Table 2. We find that DLA networks can achieve better performance than their baselines with fewer parameters. DLA-36 and ResNet-34 both use basic blocks, but DLA-36 has about 30% fewer parameters and about 1 point improvement in top-1 error rate. We usually expect diminishing returns of performances with deeper networks. Larger number of parameters can also learn the scale information in the networks with stacking structure. However, our results show that compared to ResNet-50 and ResNet-101, DLA networks can still outperform the baselines significantly with fewer parameters.
DLA vs ResNeXt shows that DLA is flexible enough to use different kind of convolutional blocks ans still have advantage in efficient parameter utilization as shown in Table 2. Our models based on the split blocks have much fewer parameters but they still have similar performance with ResNeXt models. For example, DLA-104-S has nearly the half number of parameters compared to ResNeXt-101, but the error rate difference is only 0.2%.
Compact models have received a lot of attention due to the limited hardware capability to run convolutional networks on consumer devices. We also test DLA networks with parameter number constraints to study how efficient DLA can make use of model parameters and layer outputs. We compare to SequeezeNet (Iandola et al. 2016) and MobileNet (Howard et al. 2017) in our experiments. The results are listed in Table 3. We find that DLA networks can outperform the other networks with same number of parameters. The performance is further improved with the split blocks proposed by Xie et al. (2017) are used. For a direct comparison with stacking structure, we also build and train ResNet-42-s and ResNet-54-s. They use the same convolutional blocks with DLA-48-B-s and similar number of parameters. The results show that DLA-48-B-s can still achieve much better classification accuracy.   Table 3: Comparison with other small models

Fine-grained Categorization
We follow the Inception-style scale and aspect ration augmentation (Szegedy et al. 2015), AlexNetstyle PCA-based noise (Krizhevsky et al. 2012) and the photometric distortions from (Howard 2013), in order to mitigate against over-fitting because of the limited size of fine-grained datasets. We use the same training procedure for all of fine-grained experiments. The training is performed by SGD with a mini-batch size of 64, while the learning rate starts from 0.01 and is then divided by 10 every 50 epochs, for 110 epochs in total. We use the same momentum, weight decay, mean, std, eigenvalues and eigenvectors as ImageNet Classification regardless of fine-grained datasets.   Table 5: Comparison with state-of-the-art methods on fine-grained datasets. Image classification accuracy on Birds (Wah et al. 2011), Car (Krause et al. 2013), Plane (Maji et al. 2013), Food (Bossard et al. 2014), andNABirds (Van Horn et al. 2015). Higher is better. P is the number of parameters in each model. For fair comparison, we calculate the number of parameters for 1000-way classification. V-and R-indicate the base model as VGGNet-16 and ResNet-50, respectively. We evaluate our model for fine-grained recognition on Birds (Wah et al. 2011), Car (Krause et al. 2013), Plane (Maji et al. 2013), Food (Bossard et al. 2014), andNABirds (Van Horn et al. 2015). The statistics of these datasets can be found in Table 4, while results are shown in Table 5. For fair comparison, we follow the experimental setup of Cui et al. (2017). We randomly crop 224×224 in resized 256×256 images for Bossard et al. (2014) and 448×448 in resized 512×512 for the rest of datasets, while keeping 224×224 input size for original VGGNet-16 (V-Baseline).
Most of our models are comparable to the state-of-the-art methods, without any additional annotations or specific module for fine-grained task. In particular, we establish new state-of-the-arts results on Car, Plane, Food, and NABirds datasets. In addition, our models with only several million parameters laos show very competitive performance. However, we do not get better results on Birds than the previous works, probably because it has fewer examples in each category than the other datasets do, as shown in Table 4.

Semantic Segmentation
In this section, we reports two experiments for urban scene understanding: using the CamVid dataset (Brostow et al. 2009), and the Cityscapes dataset (Cordts et al. 2016). We use the mean IoU score (Everingham et al. 2010) as evaluation metrics for both datasets. We only train our model on the training set without any other extra data. We do not use the validation set. We adopt the same optimization procedures with SGD for all the experiments. We start with learning rate 10 −3 and momentum 10 −4 . The learning rate is reduced to 10 −4 after 800 epochs for CamVid and 200 epochs for Cityscapes. At each optimization step, we sample 8 images from the dataset and a crop is sampled from from each image in the batch with 50% chance mirroring. The crop size is 400 for CamVid and 896 for Cityscapes. We didn't use any other data augmentation so that we can compare the baseline performance for all the networks.   Table 7: Performance of dilated residual networks on the Cityscapes validation set. Higher is better. Method marked with † used IDA to upsample the prediction results, while the other methods use bilinear interpolation to upsample the feature maps that are 8× smaller than input images in spatial resolution. The additional parameters for IDA upsampling is negligible. DRN-26 and DRN-42 are in DRN-C family proposed by Yu et al. (2017) On CamVid (Sturgess et al. 2009), we use 367 training images, 100 validation images, and 233 test images with 11 semantic categories for CamVid. The results could be found in Table 6. The output prediction probability maps from vanilla dilated DLA networks downsample the input images by 8. We test two methods to upsample the predictions. One use 8× bilinear interpolation and the other use LDA to upsample the prediction by 4 times followed by 2× bilinear interpolation. The results are shown in Table 6. We find that DLA-36 with IDA upsampling can beat the previous methods by more than 4 points. DLA-62-S-s is also very competitive, although it only has 1.3 million parameters. It shows that DLA can make use of the small number of images and parameters in training more efficiently than the other networks.
We also test DLA-36 on Citycapes and the results are in Table 7. We use 2, 975 training images, 500 validation images, and 1, 525 test images with 19 semantic categories for Cityscapes. DLA-36 can perform similarly as DRN-26 proposed by Yu et al. (2017) with less parameters and without degriding layers proposed in that paper. IDA upsampling can also improve the performance significantly and beat the previous baseline methods.