{'fraction_non_alphanumeric': 0.04268321909600729, 'fraction_numerical': 0.011383789483775105, 'mean_word_length': 4.666349809885932, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Global and Local Features through Gaussian Mixture Models on Image Semantic Segmentation

The semantic segmentation task aims at dense classification at the pixel-wise level. Deep models exhibited progress in tackling this task. However, one remaining problem with these approaches is the loss of spatial precision, often produced at the segmented objects' boundaries. Our proposed model addresses this problem by providing an internal structure for the feature representations while extracting a global representation that supports the former. To fit the internal structure, during training, we predict a Gaussian Mixture Model from the data, which, merged with the skip connections and the decoding stage, helps avoid wrong inductive biases. Furthermore, our results show that we can improve semantic segmentation by providing both learning representations (global and local) with a clustering behavior and combining them. Finally, we present results demonstrating our advances in Cityscapes and Synthia datasets.


I. INTRODUCTION
Humans naturally (innate way) receive and understand a large amount of information (multiple samples) [1]. Trying to replicate this process through computers with visual perception, i.e., endowing the machines the ability to see, is called computer vision [2], which is difficult due largely to variance. A vision system is required to infer objects across huge variations in pose, appearance, viewpoint, illumination, and occlusion throughout the image.
Scene understanding is one of the most challenging tasks in computer vision. It plays an essential role in many novels and future applications, e.g., autonomous driving or object detection. While humans can easily capture a scene at a glance (i.e., perceive general and specific details), it is difficult for a machine. Thus, semantic scene segmentation is one crucial step toward scene understanding, and it is one of the key challenges in computer vision. In a broader view, semantic segmentation (SS) [3] is a high-level task that makes complete scene understanding possible. The goal of scene understanding is to make machines see like humans, i.e., to have a complete understanding of visual scenes. SS aims to annotate each pixel of an image with a class label describing what this pixel represents. SS task is also called dense prediction since each pixel in the image is classified. How to extract and interpret the different levels of information details (i.e., global and local context information) is still an open problem and even more for a dense pixelwise classification. Note that in solving the SS task, this one directly influences different applications, for example, selfdrive vehicles [4], [5], segmentation on X-ray [6], detect crown on dental X-ray [7], brain tumor segmentation [8], [9], remote sensing [10]- [12], among others.
Over the last decade, computer vision has benefited from significant developments in deep learning, particularly by convolutional neural networks (CNNs) that perform remarkably on complex vision tasks such as object classification and SS. With the help of ever-growing amounts of labeled data, the deep networks can better generalize and model their context  information than the traditional methods [13]. Long et al. [14] showed that CNNs could be adapted for the segmentation task by adding upsampling layers to perform pixel-wise predictions. However, with the deep learning approach, new problems have been observed ( Fig. 1) that derive from using CNNs for the specific task of semantic segmentation. These problems are [15], [16]: (i) low-resolution obtained in the output of the CNNs, and (ii) loss of spatial precision of objects within the image. These problems are not produced by a specific operation, e.g., downsampling, but rather by a set of operations. For instance, the absence of reconstruction and refinement methods, the excessive down-sampling, the gradient vanishing, or the lack of a better extractor of feature maps [17]. Nowadays, different models [18]- [21] have addressed these problems, especially in the low resolution on the output maps. However, the loss of spatial precision problem persists and is commonly visualized at the edge of segmented objects, cf. Fig. 1. In this work, we found (see Tables 1 and 2) that to preserve spatial precision, we need mechanisms that provide general and specific details to segment, i.e., methods for better extraction, modeling, and treatment of global and local context information.
Thus, some models [15], [22], [23] use heatmap refinement with post-processing steps-e.g., conditional random field [24]-as well as models adjusting the maps from bounding boxes [25]- [27]. Although post-processing helps to have a broader-view field of objects (i.e., global features), the models lack effective global feature extractors embedded in the architecture. The encoder-decoder models [28]- [30] carry out more detailed work to adjust the heat maps by adding operations in the reconstruction, i.e., upsampling and deconvolution. These models perform local and global feature extraction at some level. However, they require a more robust combination of features than skip connections and concatenations. Some models [31]- [34] work with samples at different scales (multi-scale models) to obtain a full context of the images, i.e., global and local context information from upper and lower scales, respectively. However, multiple-size inputs make the merging process more complicated than it needs to be. In contrast, instead of resizing the inputs, models [35]- [38] increase the field of vision of the kernels (i.e., receptive field) through multiple dilated convolutions [15]. With the information from different scopes, they have the ability to tackle objects of varying sizes. However, the sampling ranges distribution (e.g., global features and context priors) cannot ensure that the information can be contained in particular ranges. That is, they have drawbacks with objects of biggersize than the convolution kernel pyramid. Finally, models [34], [39] are emerging, focusing on smart feature extraction through attention. However, performing attention in limited regions of the image shows similar behavior to the previous models, i.e., larger-size objects escape the focus of attention.
We address the loss of spatial precision problem for SS tasks by including specific structures to extract local and global features (i.e., holistic and specific features extraction blocks). Thus, we propose a Probabilistic Hourglass Gaussian Mixture Model (PHGMM), which combines Gaussian Mixture Model (GMM) [40], [41] by assuming that the data points are generated from a Mixture-of-Gaussians and a Variational AutoEncoder (VAE) [42]. Unlike the previous models, PHGMM learns two distributions by providing structures to the internal representations (i.e., latent spaces) of the SS task given an input. The first distribution extracts the global features (i.e., global context) useful for coarse SS. In contrast, the second distribution focuses on obtaining the specific features (i.e., local context) for a fine SS. Furthermore, in contrast to the previous models, we use a single sample for the multi-scale features extraction; our distributions (i.e., our learning representation) extract information independently of the objects-size to segment, combining the information in one stage robust reconstruction. Thus, we address the loss of spatial precision problem by combining the local and global features of both latent spaces and the spatial features of the encoder through skip connections and upsampling in the decoder. Our main contributions are • an end-to-end trainable deep model, PHGMM, that combines VAE and GMM for the features' internal representations; • two latent spaces modeled as a GMM to extract global and local context information that jointly improve the detection of the semantic classes (i.e., holistic and specific features extraction); • pipeline for image reconstruction by merging the different scales of context information, recovering and improving the geometric information through the decoding stage; and • a demonstration of the advantages of using coarse and fine modeling for the image information.

II. RELATED WORK
Semantic segmentation aims at dense pixel-wise classification, i.e., assign labels to each pixel in fine-grained fashion. . Given an input image, we extract local, z, and global, g, features (orange box) to improve the prediction of the semantic classes. We model the local features as GMM that fit a predicted posterior from the data (light blue box). For the global features, we use a single Gaussian to hold the holistic information of the image (green box). Note that we managed to recover the local features at the decoder stage by combining the GMMs of the latent space z with the encoder features through skip connections and deconvolution operations (purple box). Finally, the whole process is regularized by making the predictions for matching the ground truth, s, and making the local and global spaces similar to their respective priors, z , and g , respectively.
Due to this shortage of contextual information, the FCN model suffers from two main drawbacks: low-resolution output maps and a loss in spatial precision.
Existing models [15], [22], [43] present the first effort to address these problems by refining the output maps using Conditional Random Field (CRF) [24]. However, most of the models that perform post-processing steps lack local and global feature extractors that are efficient enough to discriminate the object boundaries. In contrast, instead of performing holistic post-processing, other models [25]- [27] focus on the refinement from bounding boxes, i.e., they obtain a segmentation heat map in each region of interest.
However, the transfer of information (context) remains limited. Recent works [17], [35], [36], [44] verify the importance of context information in SS. How to extract, merge, and employ this context information is the next step in SS. Thus, encoder-decoder models, such as U-Net [28], DeconvNet [29], SegNet [30], or DUNet [45], add a decoder stage (i.e., a set of deconvolution and unpooling operations) with skip connection, managing to merge, at some level, local context from low-level features and global context from high-level ones. For instance, ParseNet [46] adds global information to layers through the global average pooling operation, and FC-DenseNet [47] combines knowledge by concatenating all the features of the previous output layers. Although the encoderdecoder models show to focus on details, they still require additional detailed connections to improve their information reconstruction stage.
Although theoretically, in the layers with lower resolution (e.g., size of H/32×W/32), there is a greater receptive field; empirical experiments showed that these fields are significantly smaller and are not enough to capture global context information (i.e., global features) [36], [46]. One way to address these drawbacks is through multi-scale models [32], [34], [48] (e.g., ICNet [33] is fed with different input sizes). DeepLabv2 [15], DeepLabv3 [35], DeepLabv3++ [37], De-coupleSegNets [49] and Gated-SCNN [50] preserve the spatial size of the features maps by proposing Atrous Spatial Pyramid Pooling (ASSP). ASPP employs a set of atrous convolutions operations [51] and average pooling to capture several context information (from local to global features). However, dilated convolutions can cause grinding problems [52], [53]. It can induce a loss of local information in the models and capture irrelevant information on a large scale attention models: A different approach is presented by PSPNet [36] and ESPNet [54], operating not on the convolution kernels but on sub-regions of the feature maps generated at different levels. They use the Pyramid Pooling Module (PMM) to get global and local context information by reducing the spatial size of features. Also, HRNetv2 [55] and HRNet+OCR [56] perform multi-scale feature extraction by sharing feature maps across different branches (scales), i.e., broadcasting context information at various resolutions. Nevertheless, how and where to combine multi-scale information still represents a challenge.
Although the previous context fusion models help capture different scales' features, The relationship between objects in a global view (essential to SS) is still limited. Thus, attention mechanisms [57]- [60] can use to extract long-range contextual information. Furthermore, CCNet [44] efficiently makes a feature from any position perceive the other features, i.e., it adds contextual information in horizontal and vertical directions with sparse attention. In contrast, DANet [61] proposes two attention modules (dual attentions), focused VOLUME V, 2022 . .
. Local Space diagram. Note, the blocks and arrows represent the feature vectors and operations, respectively. We extract features at different scales through convolution operations with a sigmoid activation function. Next, we use the Global Average Pooling operation (GAP) to summarize the information and gather it into a concatenated feature vector. To model the latent space z through a GMM, we use the Fully Connected layer (FC) to extract K different Gaussian mixtures. Next, the mean of each Gaussian is used and stored in a list, and apply an FC with ReLU is to do further feature processing. Finally, we return to our initial vector size (i.e., h 32 × w 32 ). We concatenate the K Gaussians processed and perform the last convolution operation to fuse all the information from the different Gaussians. Although we compress the information using GAP to model the Gaussians distribution, the information contained in the latent space z plays an important role in adequately recovering the geometric information (background and contour of the segmented objects) through the decoder stage. on spatial features and channels. Finally, OCNet [62] extracts global and local context by grouping (by permutation), splitting, and operating each sub-region with a self-attention module (i.e., fully matrix).
Unlike previous models, PHGMM extracts local and global context information from two specific latent spaces, which we provide an adequate structure and behavior to obtain this context information precisely. In addition to providing a decoding stage capable of recovering the geometric information of the objects (background and boundary) by combining both context information (local y global) with the decoder information through skip connections, deconvolutions, and upsampling.

III. GLOBAL AND LOCAL FEATURE MODELING
In segmentation, global features can generalize entire objects or a set in a compact representation, while local features, on the other hand, compute representations for particular (more focused) parts of the image. However, due to the sensitivity to occlusion in global features or noise in local ones, it is not straightforward to combine both features reliably [2]. Consequently, we propose a model capable of merging these local and global features in a suitable manner. Our contribution comes from combining the features in two stages. First, we decode the local features representation, and then add the global ones once the image structure has been recovered in the decoder. We recover structural information, i.e., background and contour of the objects, by merging local and global features (latent spaces) with decoder features through skip connections and upsampling in the decoding stage. Thus, we obtain diverse types of contextual information, reducing overfitting and addressing the loss of spatial precision problem.
Current semantic segmentation models work by extracting features and pooling them to reduce the representation dimension. Then, while decoding the encoded features, these models pass part of the encoded information to aid the decoder in producing better features. In a way, these models use local features and share them across the encoding and decoding tasks to improve the final prediction [29]. Differently, we propose to use local and global features to improve the classes' representation by adding information to the shared knowledge between these two stages. Our proposal is to create a global model that holds the information of macro objects of the scene, while using the traditional local features, as well as the helpful combination of these and encoder features; through the decoder. We propose to use Gaussian Mixture Models to represent both spaces (i.e., global and local), since we need to hold multi-modal representations due to the different classes present in a scene (cf. Fig. 2).
Our objective is to model two latent spaces to perform a better and more robust extraction of local ( Fig. 3) and global ( Fig. 4) features and an effective way to combine them through the decoder (Fig. 5) to recover the geometric information of the objects. Thus, we propose using an endto-end model that fits and employs two latent spaces suitable for the segmentation task. In order to provide an internal structure to the learned representation (i.e., latent space), we merge VAE with ResNet-101 [63] (for an encoder-decoder stage for SS). A VAE allows us to represent the latent space through a structure, i.e., a predefined distribution. Furthermore, the ResNet-101 backbone provides robust processing (e.g., residual block) to extract the basic features that will turn into the distributions. As a result, our model obtains a local context for the discrimination of segmented objects (i.e., class-level information) as a mixture of Gaussians. The model mixes the structure of the local-global features by employing concatenation operation and upsampling throughout the deconvolution stage. In addition, the decoder stage is responsible for recovering the geometric information of the objects. Combining the context information (local-global structure features) previously extracted with the features coming from the encoder through skip connections in the different scales. Integrating these two latent spaces provides the neural network with a powerful representation (cf. Fig. 6).
The latent spaces provide complementary information that improves the semantic segmentation (cf . Tables 1 and 2).
We present an overview of our PHGMM model in Fig. 2, and detail the inference and generation processes in Sections III-A and III-B, respectively. Also, in Section IV-A, we split the description of the operations used for local and global context extraction. The local and global extraction processes are shown in Figs. 3 and 4, respectively. Fig. 5 shows that our decoder is in charge of intelligently merging the previously extracted features so that PHGMM recovers the geometric information of the segmented objects addressing the problem of loss of spatial precision (see Table. 7). Finally, in Section IV-B, we describe how our model is trained.

A. INFERENCE PROCESS
We jointly model the global information of the scene g and the local features z of the image x and assume that they are independent. We set the distribution that models this relation as is a mixture of Gaussians that models K clusters of related features that are helpful in obtaining different classes for the segmentation. The functions µ k z (·), σ k z (·), and π k (·) are the parameters of the k-th component of the mixture and depend on the given image x (see Fig. 2). All are approximated through a ResNet-101 backbone with separate projection heads that output the corresponding parameter. On the other hand, the global distribution is a single Gaussian that models a global context that holds all the image x information. The parameter functions µ g (·) The decoding diagram is fed by the previously processed feature vector (i.e., local space z). Next, the decoder takes the skip connections, deconvolution, and upsampling from the encoding. Finally, in its last layer, it concatenates the information of the different levels (i.e., multi-scale information) with the global information (i.e., global space g). Note, we call the latent space g a global representation because this space summarizes features of objects in a holistic way (i.e., a global view of the object). Also, our decoding stage is in charge of merging the different context information extracted at the different levels of our network. This smart fusion recovers the geometric information of the segmented objects. and σ g (·) are computed in a similar way as the previous parameters.
The local features z hold information about the features, while the global ones g serve as a latent variable to model holistic information from the whole scene. Additionally, due to the way we engineer the decoding stage (see Fig. 2), the global features also serve to spread the gradient back to the encoder. VOLUME V, 2022

B. GENERATIVE PROCESS
To obtain the classes y for the segmentation, we model the joint probability p(x, y, z, g) of the data, the images x and the labels y, and the latent variables, the local z and global g. We factorize the joint as p θ,φ (x, y, z, g) = p φ (y | z, g)p θ (z | x)p θ (g | x)p(x).
To implement the generative process from p(y|z, g), we use the decoder f (μ z ), whereμ z = K k=1 µ k z is the concatenation of all the means from the mixture. Later, a merge function f m (f (μ z ), µ g , f enc ) combines the partially decoded segmentations with the parameters from the global context and the encoder features, f enc , from the skip connections. Hence, the predicted labels are y = f m (f (μ z ), µ g , f enc ). For a better interpretation, we present the associated diagram in Fig. 5, in later sections.

IV. IMPLEMENTATION
In this section, we explain in detail the PHGMM architecture, i.e., local space z and global space g, as well as the operations involved in modeling latent spaces. In addition, we explain how to merge the different local-global context information and the encoder features to recover the details of the segmented objects (i.e., the geometric information). Finally, we perform the loss functions used to adjust the PHGMM parameters and show the hyperparameters used for our model.

A. ARCHITECTURE
We use a ResNet-101 as a backbone for the PHGMM model due to its good behavior, i.e., the trade-off between good feature extraction and the number of parameters used, that is, most of the residual blocks and depths of the layers. The samples that PHGMM receives as input are passed through convolution with kernel 7×7 and depth 64 to carry out the information compression, thus converting the samples in size h 4 × w 4 ×64. Next, we use a defined decoder in four groups of blocks of depth 256, 512, 1024, and 2048 respectively. Each group presents a set of 3, 4, 23, and 3 internal residual blocks. Notice that our feature vector shows a reduction in the size of ( h , and ( h 32 × w 32 ) as it passes through the decoder.
One of the issues we address in this paper is how to take advantage of this multi-scale information to improve the edges of the segmented objects and thus address the problem of the loss of spatial precision. To address this, we create local space z (Fig. 3) to model the latent space through a mixture of Gaussians, focusing on local features and global space g (Fig. 4) to extract global information through Gaussian modeling of a second latent space. Additionally, we use skip connections to pass complemental multi-scale information from the encoder to the decoder.
Thus, in local space z (see Fig. 3), we use 3×3 kernel convolutions together with a non-linear sigmoid function to compress the information of the different levels, reducing the depth from 256, 512, 1024, and 2048 for the number of classes (C). We concatenate the feature vectors to merge the multi-scale information through Global Average Pooling (GAP). To model the latent space z, we use a Fully Connected layer (FC) and produce K mixtures of Gaussians (i.e., µ k z , σ k z , π k z ). Note that the concatenation vector and the Gaussian parameters (i.e., µ, σ) have a size of 1×256 (chosen empirically). We combine the local information from the GMMs by stacking the K vectors (i.e., µ k z ×π k z ). For each item in the stack, we perform an FC with a non-linear function RELu, transforming the features of size 1×256 to 1×1×D. Then we scale the vectors achieving h 32 × w 32 ×D. Finally, we perform a non-linear combination on the concatenation vector of size h 32 × w 32 ×[K ×D], reaching a depth of 512. These feature vector locales will be used as input for our encoder. Note that even though we use GAPs in the process of converting the samples to Gaussian distributions, we call local latent space to the local space z because, from this feature vector, the geometric reconstruction is performed to recover and refine the details of the edges of segmented objects.
On the other hand, the global space g (see Fig. 4) performs the treatment of multi-scale information to a certain extent similar to the local space z, through the concatenation (of size 1×[4×C]) of feature vectors summarized by GAP operations. Next, we use two FC to model the latent space g and give it the structure of a Normal Gaussian with parameters µ g and σ g , both of size 1×256 (chosen empirically). Finally, the information is combined by taking µ g and passing it through an FC, obtaining a vector of global features of size 1×M , followed by an upscale operation achieving a vector size of h 32 × w 32 ×M . Note that we use the latent space g as a global representation because g is in charge of extracting the holistic information of the objects and then concatenates it in the last reconstruction layer of the decoder.
We merge both latent spaces (z and g), and decoder features through the decoding stage. The decoder (see Fig. 5) obtains as input the latent space z and has four groups of residual units, where each group has 3, 23, 4, and 3 residual units, all with 3×C depth and a feature vector of size h 32 × w 32 , h 16 × w 16 , h 8 × w 8 and h 4 × w 4 respectively. In each residual unit group of the decoder, our PHGMM model performs a geometric information retrieval by combining the local features coming from the GMM (latent space z) with the features extracted in the encoding through an addition operation. We transform the encoder features through nonlinear combination, i.e., convolution operations and non-linearity functions, using a 3×3 kernel with depth C. These features are passed from the different levels of the encoder to the decoder (i.e., multi-level information) with the use of skip connections. In the last layer, we concatenate the decoder's multi-scale reconstruction information through convolution operations with kernel 3×3 and a non-linearity function RELu and upscale applied to each level (i.e., group) of the decoder. Besides, a convolution operation of 3×3 with upscale to latent space g. The concatenation of all this information gives a resulting feature vector of size h 4 × w 4 ×[3×C ×M ]. Finally, for the output of PHGMM, a convolution operation with kernel 8×8 and a Softmax are applied to combine all the information and obtain our final heat map.

B. TRAINING
The target function used to train our PHGMM model L, per data sample, is a linear combination of the specific loss functions g , z , and s functions, defined by where g and z are bound to provide a useful structure (i.e., helpful embedding) for the latent spaces, and s on the loss function L focuses on dense pixel-level classification. Also, λ g,z,s are hyperparameters of each term, respectively. Note, to train with a batch of data, we aggregate the losses over it. For learning the global representation g, we need to find a relevant embedding for the segmentation task (i.e., find useful low-level features). Consequently, we use Kullback-Leibler divergence (KL) [69] to provide a structure to g w.r.t. its prior, i.e., The KL works as a regularizer over g and measures the divergence between the encoder distribution q(g | x) and the prior p(g). We specify p(g) as a standard Normal distribution p(g) = N (g; 0, I), and define q(g | x) as before (2). Our second term provides the latent space z with a clustering behavior. Instead of using a default prior, we infer a conditional p(z | x, y) from data with another net, called PostNet (see Fig. 2). This PostNet uses the ground truth labels and image data to produce the posterior distribution while training to serve as regularizer. The posterior is also a mixture of Gaussians defined by p γ (z | x, y) = K k=1 π k z * (x, y)N z; µ k z * (x, y), σ k z * (x, y) 2 I .
(5) We fit the predicted distribution with the infer one through Recall that our PHGMM model learns to densely predict the semantic classes given an input image x. Thus, the reconstruction loss function is expressed as a cross-entropy loss between the prediction y and its ground-truth y. Penalizing the pixelwise prediction for each class is not enough. Thus, we penalize the segmented objects' contours as well. We employ the loss function soft intersection-over-union [70]. So our third term, s , is where y c is our prediction of the ground truth y c for the class c. Finally, We provide an algorithm in Alg. 1 to better understand the training framework, which summarizes the PHGMM steps process.
Algorithm 1 Training the PHGMM model θEnc, φDec, γPosNet ← Initialize networks parameters repeat X ← mini-batch image from dataset Y ← mini-batch label from dataset Eq. 3 Update parameters according to gradients θEnc until static stopping criterion (epoch = 100) Number stactic of epochs

C. SETUP
The ResNet inspires PHGMM's architecture because it presents an extensive feature extraction process in the encoding stage. The encoder has four blocks of depths 256, 512, 1024, and 2048. The convolution, atrous convolution operations, and ReLU activation functions are performed within each block. Because of our limited computational resources, we reduce our input image by 1 4 . We use pooling and convolution with a stride size of two to achieve it. To produce the parameters of the latent spaces, we concatenate the output of each block. Thus, we use convolution and a Sigmoid function to adjust the depth (i.e., number of classes). Then, by mean reduction over the height and width, we obtain a feature vector of each block. We concatenate all of them and pass these through a Fully Connected layer (FC) producing our parameters (µ * or σ * ) in size 1×1×256 (empirically chosen). Note, we use different parameters (weights) in FC to infer each cluster k (i.e., µ k z , σ k z ). The same configuration is also applied for PostNet.
In the decoder, we concatenate the µ k z of every K clusters. Then, we use FC to bring our GMM vector (from z) from 1D to 2D and feed our decoder with it. The decoding stage presents operations similar to the encoder adding a bilinear interpolation operation. Moreover, it also has four blocks, all with depth equal to the number of classes. Note that in each block, the PHGMM model retrieves the geometric information (background and borders of the objects) by merging the local-context features z with the convolution operations, upscale, and the features coming from the encoder through skip connections. Finally, we add a vector of specific features from µ g (passed through FC operations and bilinear interpolation) to the decoder output. In the end, we perform two last convolution operations followed by the softmax function. Furthermore, we set our hyperparameters λ g,z,s to 1.1, 0.4, and 0.4, respectively. Finally, we use a static stopping VOLUME V, 2022 criterion when training our model. We define a static number of epochs (one hundred epochs) to stop the training.

V. EXPERIMENTS
In this section, we introduced datasets for SS used in the training and testing of our PHGMM model and the evaluation metrics used to compare the different existing models. We use the dataset with reduced classes proposed by Valada et al. [66]. It is due to our limited computational resources. To make the comparison fair (see Table 1 and 2), we train all models from scratch and each model with the same number of classes and same resolution, and use their respective hyperparameters from each model.

A. DATASETS
We evaluated several types of urban/forest scenarios datasets. They are Cityscapes [71] and Synthia [72]. Cityscapes: The dataset has 5000 samples. However, we used crop for augmenting the training set (i.e., transformations of contrast, brightness, and horizontal flip) to generate 17 500 samples. For comparison with other models, we employ the original validation set with a resolution of 768×384 (resize). To facilitate the models comparison, AdapNet++ [73] proposed the class reduction produced by combining some types of classes. Thus, we use 11 classes: sky, building, road, sidewalk, fence, vegetation, pole, car/truck/bus, traffic sign, person, rider/bicycle/motorbike, and background.
Synthia: This dataset contains realistic photo images from a virtual city. We use the original 9000 samples (7000 for training and 2000 testing) resized to 768×384 resolution. The classes of object labels are the same as the Cityscapes mentioned above label set.
the precision (Prec) is and the recall (Rec) is We assume that P i is the set of pixels predicted as the ith class, Y i is pixels set belonging to the ith class, and N is the number of classes. Besides, TP i , FP i , TN i , and FN i represent True/False Positives and True/False Negatives, respectively, for a given class i. Note, these metrics are widely used in SS We use clustering metrics to measure the latent space behavior. Consequently, we utilize the Silhouette Coefficient, where a i , b i are the mean intra-cluster and nearest-cluster distances from i, respectively. The Calinski-Harabasz Index, CHI [75] is given by where k is the number of clusters, and N is the samples, SS W and SS M are the overall within-cluster and between-cluster variances, respectively. Finally, the Davies-Bouldin Index, DBI [76] denoted by where s i is the average distance between each point of cluster i and its centroid, and d ij is the distance between cluster centroids i and j. Note, high values are better for SSI and CHI, while low ones are better for DBI.

C. COMPARISONS
We present a set of experiments (quantitative and qualitative) to compare our PHGMM model with existing models in the literature. For these, we employed Cityscape and Synthia datasets presented in the previous section. We present the quantitative results in Tables 1 and 2. We use the IoU metric (higher is better) for each class on both datasets. Note that to perform the comparison, we train all the models from scratch, with the same sample size and the same number of classes. Due to our limited computational resources, we use the sample size (i.e., 768×384) and the number of classes (i.e., 11 target classes) defined in previous works by Valada et al. [77]. We present the qualitative results in Fig. 8. The results are displayed on a set of random samples from the validation set. These figures show the inferences of the models with the best segmentation performance (i.e., IoU). We present the predictions of the different models named in the columns. These predictions show each dataset's respective color (label of classes) for Cityscape and Synthia and are displayed in the odd rows named output. The even rows named comparison VOLUME V, 2022  Fig. 7).
Remember, we conducted this research to address the loss of spatial precision in segmentation. We use Trimap [24], [70], which focuses on measuring the pixel-wise error on boundary regions of segmentation, with a distance to the boundaries given by trimap width (pixels). The plot in Fig. 7 (error curve comparison) demonstrates that our model has less loss of spatial precision compared to the literature (i.e., HRNet + OCR).
In addition, we present an ablation study (Fig. 6) on the latent space z (number of clusters K used in the GMM) and latent space g (including or excluding it from the architecture). While it may be interesting to also evaluate the global features alone, we did not perform this experiment due to the low performance that these features may produce given their location on the decoder (cf. Fig. 2). However, from these plots, we can observe that our best results are produced when using the learning representation z (local features) with the number of clusters k equal to the number of classes c in the dataset (i.e., c = k = 12). Thus, we intuit that, in an unsupervised way, each cluster of z (i.e., GMM) is modeled to obtain the best representation of local features focused on each class. Furthermore, our ablation study also shows that the latent space g is crucial to improving the final segmentation. The combination of features (z + g) produces the best predictions.
Finally, we visualize the behavior of latent space z (Fig. 9). A Mixture of Gaussian fits z through the different iterations using the multidimensional projection method t-SNE [78]. Additionally, we show the influence of z in the prediction of the segmentation. The visualization, obtained with t-SNE [78], shows that the segmentation results are improved by providing clustering behavior to the latent space z. Also, we present an efficiency comparison of the SS models (see Fig. 10) for the entire dataset and per sample. We used the Cityscapes dataset with 17 500 training samples and 500 validation samples (in our case, they are the validation samples) for these experiments. The results show the execution time of one epoch (a forward-pass over the entire dataset) in a single GPU. We executed the process five times and reported the averages of the entire training and validation dataset (left) and per sample (right), both in minutes. Our plots on training (Fig. 10) show an increase in the time required to train the PHGMM model. This increment is directly related to the use of the PostNet network and the optimization algorithm used to fit the weights in the training phase. Note that the PostNet proved robust enough to fit a Gaussian mixture model structure in the latent space. Remember, we do not use PostNet for the validation dataset, significantly reducing execution time. Finally, the previous experiments have been conducted on four Nvidia GTX Titan X GPU 12 GB graphics cards, 1 TB of RAM, and 56 Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40 GHz (multi-GPU). Also, we use Python3, Numpy, Pillow, Tensorflow 1.8, Docker, and Ubuntu operating system.
Our previous experiments show that the SS task can be improved by endowing/providing a structure to the latent space z (i.e., learning representation) in the form of a Gaussian mixture. In order to take advantage of this rich feature extraction with structure and behavior clustering, we intelligently merge these features with encoder skip connections at the decoding stage. We are achieving in this way to recover the geometric information of the segmented objects (i.e., background and boundary of the objects). In addition, it is necessary to add complementary information (latent space t) focused on the contour of the objects, to improve the boundary of the segmentation (see Fig. 6). Finally, we address the loss of spatial precision due to the combination of three factors: the PHGMM architecture (merge features in the decoder stage), the extraction of specific features given by the latent space t, and the internal structure (Gaussian mixture model) imposed on the latent space z.

VI. FUTURE WORKS
Current models extract the context (generally global) of the objects in the images and delimit the object regions. However, small regions on the edges of these objects demonstrate spatial difficulties and are lost (see Fig. 1). This loss in the edges of objects can be caused by noise, blurring, smoothing, and image resolution low, among others. This work addresses the loss of spatial precision problem in semantic segmentation. As described in previous sections, the loss of information is commonly reflected in the edge of segmented objects.
Our PHGMM model works with local and global contexts presenting improved results at the edges of segmented objects. This improvement in the results has a direct correlation with the provided (pushed) clustering behavior in the latent space, see Figs. 7 and 9. In addition, this correlation is influenced by the number of clusters used in the clustering algorithm in the training stage, shown in Fig. 6. The following research step would be to increase the number of clusters through hierarchical clustering. In this way, we would group objects semantically at a higher level, and break them down into other sub-classes down in the hierarchy. Also, to accomplish this model idea, we would need a larger number of classes.

VII. CONCLUSION
In this work, we show that by endowing the latent spaces (z and g) with clustering behavior and providing them with a structural representation (i.e., GMM for z and Normal distribution for g), we improved the results of the SS task. This improvement is mainly due to the combination (produced in the decoding stage) of low-level features (from global-context information in g) and high-level ones (from local-context information in z). The combination of features is performed in the encoder stage with the latent spaces z, g, and the multi-scale features from the encoder in order to recover the geometric information. Furthermore, our comparative results show that more details can be extracted from PHGMM by setting the number of clusters equal to the number of classes. The improvements are produced at the segmented objects' boundaries, thus addressing the loss of spatial precision problem.