{'fraction_non_alphanumeric': 0.03457987237531639, 'fraction_numerical': 0.009268831770703362, 'mean_word_length': 4.913153456998313, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MorphPool: Efficient Non-linear Pooling&Unpooling in CNNs

Pooling is essentially an operation from the field of Mathematical Morphology, with max pooling as a limited special case. The more general setting of MorphPooling greatly extends the tool set for building neural networks. In addition to pooling operations, encoder-decoder networks used for pixel-level predictions also require unpooling. It is common to combine unpooling with convolution or deconvolution for up-sampling. However, using its morphological properties, unpooling can be generalised and improved. Extensive experimentation on two tasks and three large-scale datasets shows that morphological pooling and unpooling lead to improved predictive performance at much reduced parameter counts.


Introduction
Contemporary deep learning architectures exploit pooling operations for two reasons: to filter impactful activation values from feature maps, and to reduce spatial feature size [28]. The most used pooling operation is the max pool, which is used in nearly all common network architectures such as ResNet [14], VGGNet [32], and DenseNet [16]. These network architectures can be applied to pixel-level prediction tasks, such as semantic segmentation. To do so, inputs are down-sampled to a set of latent features of small spatial size, after which they are up-sampled to full resolution again. Up-sampling from pooled feature sets most often happens with a combination of unpooling and deconvolution [41,42] and is used in seminal works such as [3,22,26].
As will be shown in this paper, down-sampling using max pooling can be formalised and improved using mathematical morphology, the mathematics of contact. Ever since the works of Serra [29], the underlying algebraic structure of data that is acquired using probing contact (e.g. LiDAR and radar) has been known to the computer vision community [5,11,25,33]. It is different from the algebra of linear diffusion that is used to build convolutional neural networks (CNNs). Rather than just considering pooling, it would sensible to formalise pooling and unpooling in CNNs as fully morphological operations: in any encoding layer, pooling is a form of down-sampling and a decoder layer will ultimately need to up-sample. The task of down-sampling is one of morphological sampling and the task of up-sampling is in fact one of morphological interpolation.
The connection between morphology and pooling has been noted in the literature: concurrently with this paper, [1] remarks that max pooling and morphological operations indeed share commonalities, although they do not further explore the idea. [10] proposes a morphological alternative to pooling as a weighted combination of morphological dilation and erosion, and calls it morphological pooling. Both these papers do not formalise the pooling as morphological, nor do they treat up-sampling as the morphological semi-inverse of pooling. In this article, it is shown how pooling is of an essentially morphological nature (dilation or erosion), and that there is then a naturally associated form of up-sampling (unpooling), dictated by the tropical algebra of morphology.
The contributions of this paper are: • A formalisation of max pooling showing that it is an unparameterised non-overlapping special case of the morphological dilation. • A fully morphological up-sampling procedure that can replace the semi-morphological unpooling-deconvolution scheme that is commonly used in neural networks. • Extensive experimentation to show morphological pooling and unpooling consistently outperform other down and up-sampling schemes at highly reduced parameter counts, including efficient CUDA implementations. This effect is most pronounced on data that is of an inherently geometric nature, such as depth images.

Background
Pooling Convolutional networks employ sequential convolution layers to obtain an expressive set of features for regression or classification. Pooling layers are used to force the network to regress high-quality features, and there exist many pooling schemes (for a review, refer to [12]). The most common pooling scheme is the max pool. Max pooling is used to obtain maximum-amplitude coefficients at intervals while reducing spatial feature sizes [6,38], and to achieve invariance to small distortions from noise [17]. It is shown to be a very efficient procedure to encode meaningful features, often outperforming mean pooling (e.g. [17,40]). In [4], it is argued that max pooling is more robust to high clutter noise in class separation than mean pooling.
Unpooling While pooling is used to down-sample and refine features, up-sampling may be necessary to obtain predictions at original resolution. Up-sampling can naturally happen in a variety of ways, but of interest is the inverse of pooling. [41,42] are the first to define an inverted pooling operation using a combination of deconvolution and 3D reverse max pooling. Here a set of switches records the location of max values in the pooling step, that are then used to place back latent features at higher resolution; remaining elements are set to zero. The unpooling operation is interleaved with deconvolution, or rather transposed convolution, to obtain dense feature maps (i.e. infilling). The main advantage of deconvolution is that it does not employ any predefined interpolation scheme, but can learn the interpolation parameters 1 .
There exist many examples in literature of this unpooling-infilling scheme for up-sampling, most notably [3,22]. First, [22] introduce Deconvolutional Neural Networks. Their network interleaves 2D unpooling layers with deconvolutions, since it is argued that low-level visual features capture shape details and max pooling has been reported to recover shape well [19]. Second, [3] introduce SegNet, which interleaves 2D unpooling and convolution. The main motivation is that unpooling results in more sharply delineated semantic boundary predictions. This strategy is also employed in [20], who solve a semantic segmentation task as well.
While [3,22] employ a deconvolution and a convolution for infilling respectively, it is worth noting that the term deconvolution is misleading: it is not the deconvolution in the mathematical sense, but rather the convolution with flipped kernel (correlation). Since parameters are freely learned in networks (and setting the stride similarly for either operation) they end up being equivalent.
Variations [9] and improvements [18,39] of the unpooling-infilling scheme exist, but the (de)convolution operation remains necessary to deal with sparsity. Whereas pooling is parameterless, the infilling stage is heavily parameterised by (de)convolution to overcome this issue. It will now be shown that this is not necessary.

Method
In this section, max pooling will be shown to be a special case of the morphological dilation. Note that the same could be done for min pooling through erosion, by morphological duality [30]. First, a common notation is introduced for neural networks and individual (linear or morphological) layers within the network. A neural network f is a composite function f (x) = f L • · · · • f l • · · · • f 1 (x) of L layers. Let y n = f (x n ) be the output of the network function. The network is tasked with fitting a (sample, target) dataset D = {(x n , t n )} N n=0 . In this case, training the network is the minimisation of the energy function E(t n , y n ) ∀n, where t n is the target output. The input to a specific layer is denoted f − , where the subscript l is dropped for readability. The output to that layer is denoted f + , and the parameters of the layer are h.

Morphological Pooling
Max pooling is a discrete operator that takes a non-overlapping patch-based maximum for any given discrete input map f (x). It is shown below that max pooling is a non-overlapping, non-parameterised special case of the morphological dilation. The morphological dilation is defined on the semi-ring {R −∞ , , +} where denotes the supremum operation and + is addition. This algebraic system extends the set of real numbers R with minus infinity: A layer input signal f − : R D → R −∞ indexed by indicator variable x, and a structuring element h : R D → R −∞ indexed by indicator variable z are combined to produce the morphological dilation as the layer output signal f + : On the other hand, max pooling can be written as where f ∨ is the down-sampled layer output (similar to f + in Equation 1), f ∧ is the layer input at high resolution, s is a predefined stride (usually 2), and z is an indicator set to capture the patch over which the maximum is taken (also usually of size 2). The spatial dimensions are thus reduced by stride s.
Let x ∨ ∈ x be the output locations on the down-sampled output signal f ∨ . In that case, x ∧ is the set of input locations scaled by the stride s, that is Also, drop the assumption of discreteness to adopt the continuous supremum operator . Then, we can rewrite Equation 2 to In mathematical morphology, the structuring element h is regarded as a shape that dilates or erodes a signal. The simplest structuring element is the flat structuring element that can be thought of as a flat disc Z centred around the origin: Since h flat is symmetric, the z indices could be substituted by −z. Thus, Equation 3 becomes From Equation 5, it is clear that the max pool is in fact a dilation with a flat structuring element, and the ∨, ∧-notation for the signals f ∨ , f ∧ and indicator sets x ∨ , x ∧ is to account for striding the input by stride s. A schematic overview of pooling by dilation with flat structuring element is given in Figure 1.
While flat structuring elements are often used in grey-value morphology, there are more interesting structuring elements. One example is the parabolic structuring element [36]: Using such a structuring element is a morphological weighting, where pixels closer to the centre affect the maximum more depending on the scale σ (and algebraically it is the natural counterpart of Gaussian weighting in convolution [36]). Using parabolic h, a morphological scale space can be constructed [15,35]. Scale space is the natural way of dissecting object structure at different scales, and it provides clear intuition for why re-sampling signals can be done by dilation. These parabolic structuring elements can for example also be used to build scale equivariant networks [27,37]. The flat and parabolic structuring elements are members of a larger set of parameterised morphological operations; the set of parameterised operations generalise max pooling.
Lastly, note that convolution and dilation are logarithmically related in the sense that convolution uses a multiplication-addition scheme where dilation uses an addition-supremum (left) denotes pooling, which is a dilation with a flat structuring element of size 3x3 combined with sub-sampling resulting in f ∨ (x ∨ ) and provenance z ∨ . The image is divided in four 3x3 patches; (right) shows the two-step process of unpooling, which uses the provenance to place the maximum elements at the correct locations resulting in f ∧ (x ∧ ); then a 5x5 flat structuring element should be used to morphologically interpolate the maxima yielding the full resolution map f ∧ (x).
scheme. However, convolution on images acts on spatial as well as channel dimensions; grey-value dilation could as well, but pooling is a purely spatial operation. Therefore, the discussion is limited to purely spatial h, omitting channels dimensions from the dilation operation.

Morphological Unpooling
Morphological unpooling should invert, as much as possible, the pooling operation and upsample back to original scale. Full inversion is not possible, because of non-maximum suppression of the dilation operation. Morphological unpooling is a two-stage process. The first stage is morphological provenance mapping [35] (syntactically similar to the morphological derivative introduced in [13]). For purposes of unpooling, z ∨ from the pooling stage can be thought of as provenance: recorded locations at which to place back values during up-sampling. That is: Or, the locations x ∨ on the down-sampled signal f ∨ are transferred to the regularly sampled locations x ∧ compensated by the provenance of the maxima that were stored in z ∨ -see The values set to −∞ at the other locations x, however, cannot be used in training a neural network.
Since the values f ∨ (x ∨ ) at locations x ∨ were maxima in the pooling step, the undefined values at all x must be upper bounded by f ∨ (x ∨ ) and h when unpooling. Consequently, the second stage is the application of a morphological dilation that retains the same spatial dimensions to morphologically interpolate between the relocated maxima at x ∧ − z ∨ : where w is a scaled indicator set of s times the size of z to perform gap filling in f ∧ (x ∧ − z ∨ ). As a result, the output f ∧ (x) is guaranteed not to contain values at −∞; in stead the values have been upper bounded by the maximum Figure 1 for an example with a flat structuring element h.
In summary, morphological unpooling is a combination of (1) a strided morphological derivative to up-sample the spatial features and (2) a dilation with increased kernel size to suppress unknown values. Again, it is possible to parameterise h of the dilation in the second stage (Equation 8) to make full use of concepts from morphological scale space and morphological interpolation. The full procedure of generalised morphological pooling & unpooling is called MorphPool.

Experiments
MorphPool is evaluated on semantic segmentation and depth auto-encoding on NYUv2 [31], SUN-RGBD [43], and Stanford 2D-3D-Semantics Dataset (2D-3D-S) [2]. Morphological unpooling is compared to the standard unpooling-infilling scheme, and to linear sampling and interpolation such as combinations of (de)convolutions. To this end, DownUpNet is introduced: the simplest encoder-decoder architecture that has variable down and up-sampling blocks. The aim of the paper is a proof-of-principle, for which a simple architecture like DownUpNet is sufficient; its goal is not to compete with specialised segmentation networks to achieve state-of-the-art results. It is hypothesised that morphological operations perform better at depth data, since depth data is inherently non-linear at geometric edges. In contrast, (de)convolution does not allow for processing 3D spatial information between pixel neighbours, and is sensitive to occlusion. Implementation details are given below, but more details are found in Supplementary Material A. Additionally, results in this paper are further supported by Supplementary Material B.
Down & Up-sampling MorphPool is compared to a regular pooling & unpooling baseline like e.g. in [3,22]. In those networks, unpooling is followed by (de)convolution to deal with the sparsity that unpooling introduces. Another baseline is a linear sampling method (common in ResNet [14] architectures), where down-sampling is a strided convolution and up-sampling is bilinear interpolation followed by a convolution layer. Both methods introduce high numbers of parameters due to the (de)convolutions. More specifically, they introduce C 2 × K 2 per layer, where C are the channels and K is the kernel size. This makes the sampling procedures potentially expressive at the cost of memory. The morphological pooling & unpooling with flat structuring elements however, introduce no additional parameters. When the morphological operations are parameterised parabolically, C parameters per layer are introduced. Finally, for structuring elements that are parameterised at each spatial location -the general setting-C × K 2 parameters per layer are introduced. It will be shown that additional parameters of (de)convolution are redundant.
DownUpNet To fairly compare between down and up-sampling schemes, DownUpNet is introduced. DownUpNet down-samples 5 times, for a total reduction of spatial resolution of Figure 2: The DownUpNet architecture, which is a simple encoder-decoder. Each downsampling operation reduces spatial dimensions S by 2, for a total of 2 5 . The downsampling operations are preceded by convolution including batch normalisation and ReLU non-linearity. The up-sampling operations are followed by the same type of block. Because of this (and because depth data has more sensor noise and infilling artefacts) networks trained with depth input start with a learning rate λ = 5e−4 whereas RGB networks start with λ = 5e−3. Both learning rates exponentially decay to end at 2% of the initial λ . For training, random crops of size 384×384 are used for NYUv2 and SUN-RGBD, crops of 512 × 512 are used for 2D-3D-S. During testing, centre crops are used as close to full resolution as possible, while retaining a resolution that is divisible by 2 5 .

Semantic Segmentation on Depth
First, semantic segmentation experiments are performed on depth input. Morphological operations are expected to perform well, because depth maps are inherently non-linear in a morphological manner. That is, morphology allows data to be probed by structuring el-  ements in space; depth is a geometric modality, not a visual one. Results are shown for 2D-3D-S in Table 1, and for SUN-RGBD and NYUv2 in Table 2. Performance is measured in mean Intersection over Union (mIoU), pixel accuracy, and a boundary F1-score [8] to measure performance at semantic edges. From the results, note: • Standard unpooling is outperformed significantly by flat morphological unpooling; introduction of sparse features before (de)convolution hurts performance significantly. • It is expected that for a pooling stride of 2, the minimum unpooling with which to fill the features completely is 3. Moreover, with a pooling stride of 3, an unpooling stride of 5 is required. Both are confirmed. • Performing strided convolution as down-sampling and bilinear interpolation in combination with convolution for up-sampling does not match performance of morphological sampling on depth data. The addition of two full convolution layers, one for down-sampling and one for up-sampling introduces many parameters to the network. When flat morphological operations are used, there are no additional parameters. • For these runs, there is a numerical difference between using convolution and deconvolution. It is expected that these differences fall within a margin due to initialisation; for other experiments, the differences are not so obvious. Convolution and deconvolution should in principle be equivalent. Therefore, only convolution will be used in reporting results from now on. The complete results can be verified in Supp. B. • Morphological pooling & pooling perform best at semantic boundaries, denoted by bfscore. This is in line with the conclusions from [3,22], although a fully morphological sampling outperforms those methods.
Besides the results listed above, initial experiments also indicated that mixing linear and nonlinear sampling (e.g. down-sampling by standard pooling and up-sampling by convolution) had no practical benefits. With the natural non-linearities of morphology available, mixing linear and non-linear operations does not seem worthwhile.  Parameterisation of the Structuring Element Besides using a purely flat structuring element (SE) from Equation 4, it also also possible to parameterise MorphPool. This could be done using parabolic SEs from Equation 6 or by a general parameterised SE: each value on the discrete kernel then has its own parameter that can be freely adjusted. Results are listed in Table 3 and indicate parameterising the structuring elements is helpful. General parameterisation of the SEs performs best, although it also introduces most parameters. Note that compared to introducing an additional convolution layer (i.e. millions of parameters), introducing thousands of parameters for the structuring elements is much more computationally reasonable. General parameterisation of the SEs is used for further experiments.
It is possible to compare networks that have the same number of parameters as well. For this, depth-wise convolutions [7] are used instead of full convolutions; depth-wise convolutions only operate along spatial dimensions. Results for depth input on the SUN-RGBD dataset are listed in Table 4. Note especially the gray cells indicating a morphological and linear setting with the same number of parameters. MorphPool still outperforms the linear setting.

Semantic Segmentation on RGB
Semantic segmentation is also performed on RGB input (see Table 5). It can be concluded that, again, regular pooling and unpooling yields worse performance than generalised morphological pooling and unpooling on semantic segmentation. Now, convolution and interpolation perform on par with morphology. Note, however, that the comparison is not completely fair due to the increased number of parameters convolution has available. Interpolation networks have 50% more parameters than morphological methods. Like for semantic segmentation using depth, morphological pooling & pooling for RGB input perform best at semantic boundaries. Performance of morphological pooling and unpooling on depth-based segmentation outperforms RGB-based segmentation. Compared to depth, it is understandable that RGB signals are less easily encoded using mathematical morphology: RGB does not just express geometry but also illumination, which has aspects that are well described

Depth Auto-encoding
As a final experiment, a depth auto-encoding task is solved; results are shown in Table 6. Performance is measured in Absolute Relative Distance (ARD), Root Mean Squared Error (RMS), and an accuracy δ t of predictions within a threshold t = 1.25. The results show substantial improvement, confirming depth is a modality that is very well suited to the use of morphological operations.

Conclusions
In this paper, max pooling has been formalised as a non-parameterised non-overlapping special case of the more general morphological dilation. In view of this, the unpooling-infilling scheme can be replaced by a generalised morphological operation. The full procedure is called MorphPool. Experiments on two tasks and three datasets show that the proposed method outperforms other sampling schemes at much reduced parameter counts. The effect is most pronounced on depth data, which confirms the expectation that mathematical morphology may be the natural language to process such non-linear geometry. Given the success of morphological operations for processing depth, future research could investigate whether fully morphological encoders may be used in multi-modal networks. In that case, parallel treatment of linear and morphological features has to explored. [42] Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 international conference on computer vision, pages 2018-2025. IEEE, 2011.

Supplementary Material A: Implementation Details
Network Architecture Using the DownUpNet architecture, the quality of semantic predictions as a result of different sampling schemes is tested. A schematic overview of the network architecture is shown in Figure 2. There are three choices for down-sampling operations: • Convolution with stride 2 and kernel size 3. This operations introduces C 2 × K 2 additional parameters per down-sampling layer, where C is the number of channels of the layer and K the kernel size. • Max Pool with a non-overlapping kernel of size 2 at stride 2.
• Generalised Morphological Pooling, with variable kernel size. The pooling operation can be flat (similar to max pooling) or be parameterized with any structuring element.
In the case of parabolic structuring elements, this layer introduces C parameters. And in the freely parameterized case, this introduced C × K 2 parameters.
There are three choices for up-sampling operations: • Sparse up-sampling at rate 2, followed by bilinear interpolation and convolution. This operation introduces C 2 × K 2 additional parameters per up-sampling layer. • Unpooling followed by deconvolution as introduced in [42] and used in [22]. This operation introduces C 2 × K 2 additional parameters per up-sampling layer. • Morphological unpooling as described in Section 3.2, which unpools by provenance followed by a morphological dilation. In case of a flat structuring element, this is a parameterless layer. In case of parabolic parameterisation, this introduces C parameters. And in the case of a freely parameterized dilation, this layer introduces C × K 2 additional parameters.
Note especially that morphological pooling and unpooling (i.e. MorphPool) introduce hardly any additional parameters. Even the general (i.e. freely) parameterised dilations introduce a factor C (i.e. the number of channels) fewer parameters than full convolutions in the other sampling operations do.
In the network, all convolutions have a kernel size of 3, and are padded to retain resolution. When training on 2D-3D-S, there are two convolution blocks (including batch normalisation and ReLU non-linearity) before and after each sampling operation. This is because the 2D-3D-S is much larger than NYUv2 and SUN-RGBD and initial experimentation showed significantly improved performance of this architecture using a larger network for all methods.

Training details
All methods are implemented in PyTorch, with the exception of the morphological operations which are implemented as C++/CUDA extensions. The extensions are compiled using g++ and CUDA11. All networks are trained using SGD with Nesterov Momentum [34], with a learning rate that decays exponentially by a scalar γ. Based on initial experimentation, for consistent performance this γ can be set such that the initial learning rate λ decays to at 2% of the initial λ , according to γ = epochs 2 100 with epochs the number of training epochs, and epochs ≥ 1. Table 7: 2D-3D-S Segmentation on Depth Results. The first column denotes the sampling method, the second through fourth column denotes the kernel size with which the Pooling, Unpooling, and Convolution scheme worked. For each method of post-processing (None, Convolution, Deconvolution) the up-sampled feature volume, the number of parameters and performance is given. The high-lighted gray cells denote the standard procedures for up and down-sampling: Unpooling followed by (de)convolution, bilinear interpolation followed by convolution. In addition, General MorphPool denotes general parameterised morphological structuring elements. The bold-faced results indicate best performance per column. RGB images are normalised with training set statistics to follow a zero-mean Gaussian, as is common for neural networks. Depth images however are not normalised, and are unscaled depth values in meters. Depth data in general has more sensor noise and infilling artefacts.
In combination with the unscaled data, this led to less stable learning at similar λ to RGB images. Therefore, networks trained with depth input start with a learning rat λ = 5e−4, whereas RGB input starts with λ = 5e−3.

Supplementary Material B: Supporting Results
In this section, all supporting results for the main experiments section are given. All results are reported over a variety of tables. For semantic segmentation on Depth, see Table 7 for 2D-3D-S, Table 8 for SUN-RGBD, and Table 9 for NYUv2. For semantic segmentation on RGB, see Table 10 for 2D-3D-S, Table 11 for SUN-RGBD, and Table 12 for NYUv2. And finally, for depth auto-encoding, see Table 13.
Additionally, depth-wise convolutions can be used to equalise the number of parameters available to networks making use of morphological and linear sampling. On top of the results on depth data for SUN-RGBD in the main text, results are listed for depth input on NYU in Table 14, and for RGB input in Table 15 and Table 16. In general, for networks with the same number of parameters MorphPool outperforms its linear counterpart. Again, the effect is most pronounced for depth data.