{'fraction_non_alphanumeric': 0.03255096018522722, 'fraction_numerical': 0.012666273210151178, 'mean_word_length': 4.842970822281167, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AltUB: Alternating Training Method to Update Base Distribution of Normalizing Flow for Anomaly Detection

Unsupervised anomaly detection is coming into the spotlight these days in various practical domains due to the limited amount of anomaly data. One of the major approaches for it is a normalizing flow which pursues the invertible transformation of a complex distribution as images into an easy distribution as N(0, I). In fact, algorithms based on normalizing flow like FastFlow and CFLOW-AD establish state-of-the-art performance on unsupervised anomaly detection tasks. Nevertheless, we investigate these algorithms convert normal images into not N(0, I) as their destination, but an arbitrary normal distribution. Moreover, their performances are often unstable, which is highly critical for unsupervised tasks because data for validation are not provided. To break through these observations, we propose a simple solution AltUB which introduces alternating training to update the base distribution of normalizing flow for anomaly detection. AltUB effectively improves the stability of performance of normalizing flow. Furthermore, our method achieves the new state-of-the-art performance of the anomaly segmentation task on the MVTec AD dataset with 98.8% AUROC.


Introduction
Automated detection of anomalies has become an important area of computer vision in a variety of practical domains, including industrial (Bergmann et al. 2019), medical (Zhou et al. 2020) field, and autonomous driving systems (Jung et al. 2021). The essential goal of anomaly detection (AD) is to classify whether an object is normal or abnormal, and localize the regions of an anomaly if the object is classified as an anomaly . However, it is challenging to obtain a large number of abnormal images in real-world problems.
An unsupervised anomaly detection task has been introduced to address this imbalance in the dataset due to the lack of defected samples. To do so, it aims at detecting defection by using only the non-defected samples. That is why many recent approaches for unsupervised anomaly detection try to obtain representations of normal data and detect anomalies by comparing them with test samples' representations.
Learning representations of normal data in an unsupervised anomaly detection task is closely related to the goal of normalizing flow: the model learns invertible mapping of a complex distribution of normal samples into a simple distribution such as the normal distribution. The original usages of the normalizing flow are density estimation and generating data efficiently using invertible mappings (Dinh, Sohl-Dickstein, and Bengio 2017;Kingma and Dhariwal 2018), whereas some recent approaches (Rudolph, Wandt, and Rosenhahn 2021;Yu et al. 2021;Gudovskiy, Ishizaka, and Kozuka 2022) begin to utilize the normalizing flow to tackle the unsupervised anomaly detection tasks. In particular, they have achieved state-of-the-art performance on the MVTec AD (Bergmann et al. 2019) of the industrial domain.
Despite the success of normalizing flow-based anomaly detection models, they have a limitation for the unsupervised task: performances are unstable in many cases. Specifically, their test performances often fluctuate while training the model for a long time (e.g. 200 or 400 epochs). This drawback will be critical in every practical domain because one must use the unsupervised trained models that have been trained sufficiently. One possible reason is the limited expressive power of normalizing flow with the fixed base (prior) distribution. Current approaches as Rudolph, Wandt, and Rosenhahn (2021); Yu et al. (2021); Gudovskiy, Ishizaka, and Kozuka (2022) fix the base distribution of normalizing flow as N (0, I). However, as depicted in Figures 1 (b) and (c), most outputs of the model do not follow N (0, I), while they utilize the likelihoods in the base distribution as a score function: where C is an index set for output channels. Even the score function requires an assumption for the theoretical validity that normal samples are transformed to the base distribution while training, it fails for the models to learn N (0, I). This phenomenon occurs pervasively for normalizing flow-based anomaly detection models. To tackle this problem by considering the reason mentioned above, we propose an algorithm called the Alternating Training method to Update Base distribution (AltUB) that performs better and learns more stably with normalizing flow-based anomaly detection models. Our method introduces alternating updates to learn a base distribution more actively and to allow the model to adapt to  (Shapiro and Wilk 1965) applied to values of each spatial location of each channel among outputs from normal images. The distribution with a p-value > 0.05 is usually considered normal. (b) The result of the Kolmogorov-Smirnov test (KS-test) (Massey 1951) applied to the same ones with (a). p-value > 0.05 implies the distribution is highly likely to be N (0, 1). (c) Visualization of one of the distributions. Ideally, it should follow N (0, 1). However its distribution is roughly N (2, 0.5). Figure 2: The training process of AD flow+AltUB. The shaded (Flow, µ, Σ) are trainable, and the two backward lines (blue solid line and red dotted line) indicate the alternating updates. While training, the model learns so that likelihood is high and the anomaly score is low for normal samples.

AltUB
the changed base distribution gradually. This simple but effective updating scheme improves the expressive power of the normalizing flow, especially for anomaly detection.
To verify the stability of our method under the unsupervised anomaly detection task, we measure the average performance for some training epoch intervals as well as the best performance. In the experimental results, our method improves the stability of normalizing flow-based anomaly detection models. In addition, our method achieves stateof-the-art performance on anomaly segmentation of both MVTec AD and BTAD datasets. These results support that our method is learning-stable and performing better.
Contributions In summary, we propose AltUB for the anomaly detection tasks with the following contributions.
• We investigate that normalizing flow models for anomaly detection commonly fail to transform normal images into N (0, I). • We suggest the update of the base distribution can solve the above defect, and propose a proper method to update the base distribution: Alternating training. • Our model achieves state-of-the-art performance on anomaly detection of MVTec AD and BTAD datasets.
2 Related work 2.1 Normalizing flow-based anomaly detection models Unsupervised anomaly detection models based on normalizing flow have shown high performance on various practical tasks (Rudolph, Wandt, and Rosenhahn 2021;Yu et al. 2021;Gudovskiy, Ishizaka, and Kozuka 2022). Normalizing flowbased models first obtain representations of normal images from a pre-trained feature extractor. They utilize the representations to learn a distribution of normal data. Because of their simple yet powerful idea of invertible mappings, they effectively describe the distribution. To detect anomalies, they estimate the likelihood of the base distribution of test samples. However, their expressive power is not enough as shown in Figure 1. It might cause of their low stability on performance while learning i.e. performance decreases as learning progresses. In this paper, we suggest a simple method to increase the expressive power of normal data to improve the AUROC score. (1 < 16 < 8 < 4 < 2) (c) The training loss measured during the training. (16 < 8 < 4 < 2 < 1) gressive prior by using ConvLSTMs to increase the expressive power of split coupling layers.
However, these approaches are not appropriate for anomaly detection tasks for the following reasons. First, the output seems to follow not only channel-wise but also spatial dimensional normal distribution. Second, the layers can't distinguish normal images and abnormal ones because these layers train not parameters of distribution but the way to get them from inputs. Note that reflection of the sample statistics from anomaly disrupts the detection.
Nevertheless, a methodology specially designed to train the distribution for anomaly detection has not been suggested yet. To address issues, we propose a proper method to update the base distribution for anomaly detection tasks.
3 Preliminary: Normalizing flow 3.1 Notation θ and ψ represent the parameters of the normalizing flow and that of the base distribution, respectively. We use p(·) as a distribution and the probability density function of the distribution. p Z (·; ψ) indicates the base distribution and p X (·; θ, ψ) denotes the distribution induced from the base distribution by normalizing flow. In addition, p * X (·) is the data distribution that the model aims to learn, e.g. distribution of images.

Fitting methodology
The idea of normalizing flow is from a change of random variables as This implies that a complex distribution p X (·; θ, ψ) can be factorized to simple and known distribution p Z (·; ψ) using invertible mapping f θ : X → Z. In general, the normalizing flow stack the invertible neural layers, f θ = f θ1 • f θ2 • · · · • f θn to form the invertible mapping, where f θi is the layer of neural network. To find the correct invertible mapping f θ with respect to the Z and X, we can use maximum likelihood estimation: estimating the parameters of distribution with random samples from the distribution. In particular, maximizing the likelihood can be done by maximizing log-likelihood, which is (3) This objective function is based on the fact that maximizing the log-likelihood is equivalent to minimizing Monte Carlo (Papamakarios et al. 2021).

AltUB
In this section, we introduce AltUB, an effective method to update the base distribution of normalizing flow to enhance the expressive power of the flow and increase stability in detecting anomalies. We first state the limitations of prior normalizing flow-based works for anomaly detection, which affect the stability. Then, we propose our method.

Limitations of Non-trainable base distribution of normalizing flow
Normalizing flow-based anomaly detection models (Rudolph, Wandt, and Rosenhahn 2021;Yu et al. 2021;Gudovskiy, Ishizaka, and Kozuka 2022) define normalizing flow f θ : X → Z as a mapping from extracted features of images to simple, fixed distribution. e.g. N (0, I). The normalizing flow architecture is used for learning a distribution of representation of normal samples to detect out-of-distribution samples. However, Figure 1 shows that the models seem to transform normal images into not N (0, I) as their purpose, but into several different normal distributions of CHW dimensions due to their low expressive power. The most simple way to enhance the expressive power is to stack more layers. In fact, Komologorov-Smirnov statistics and the train loss(negative log-likelihood) decrease as flow gets deeper as shown in Figure 3 (a) and Figure 3 (b). Nevertheless, the average of mean square tends to increase while adding more layers for shallow depth, which indicates the phenomenon of Figure 1 (mean-shift) occurs. We guess the reason as follows: while an arbitrary distribution is transformed toward N (0, 1) along multiple layers of flow, the conversion into the normal distribution takes precedence over the proper adjustment of specific parameters as mean and variance. Figure 3 (a) represents that the mean-shift decreases in the deeper depth than 8. We are able to easily figure out that the mean-shift phenomenon can be solved by increasing expressive power. However, stacking layers may not lead to higher performance (see Figure 3 (c)) because of overfitting. In the other words, stacking more and more layers may disrupt learning the distribution of normal samples, even if it increases expressive power. Therefore, the flow model with non-trainable base distribution cannot break through the mean-shift simply.

Trainable base distribution of normalizing flow
As Figure 1 and Section 4.1 show, a method that increases expressive power with a few layers is required to match the model output and score function. This method may improve the performance by using the right score function to find anomalies.
Meanwhile, (Papamakarios, Pavlakou, and Murray 2017) proves that the following equation holds: where p * X (·) is a target distribution in X and p * Z (·; θ) is the induced from p * X (·) by normalizing flow f θ . The equation 4 means that the effective update of p Z (z; ψ) toward p * Z (z; φ) leads p X (x; θ) to be similar as p * X (x). Therefore, we can increase the expressive power of models by updating the base distribution. Based on this fact, AltUB assumes the base distribution of the models to be N (µ, Σ), i.e. normal distribution with learnable parameters of CHW dimensions.
In our method, along with the parameters of normalizing flow θ, the parameters of base distribution ψ are also trained with stochastic gradient-based methods using the gradient of negative log-likelihood (loss) as follows: where L(θ, ψ) = − log p X (x; θ, ψ) is the negative loglikelihood of samples from X.
As an anomaly score of a test sample with C channels, we use the likelihoods in the base distribution: Because we obtain the appropriate parameters of the base distribution from AltUB, our score function will detect the anomalies more precisely.

The method to train base distribution
When we assume the base distribution as N (µ, Σ), the gradient of negative log-likelihood of Equation 5 becomes as follows: The loss function consists of subtraction, hence the gradients will be small. However, we observe that some outputs like Figure 1 (c) are quite far from N (0, 1). Therefore, the parameters of the base distribution are not able to be trained effectively with the original learning rate of the main models, e.g. 0.001 of FastFlow. In addition, because the normalizing flow model learns slowly to find the distribution of representations of normal images, a consistently high learning rate for parameters of base distribution will harm the model. To train the base distribution well yet not harm the model to learn, the alternating training method is introduced. In general, a normalizing flow model and the base distribution are trained together with the original optimizer and learning rate η 1 . But only the base distribution is trained with the larger learning rate η 2 using an SGD optimizer once in a freezing interval. The detail is depicted in Algorithm 1.
Due to the high dimensionality of the space for base distribution, AltUB may suffer from computational complexity when using Σ. This is because of the computational complexity to calculate Σ −1 in the gradients (about O(n 3 )). Therefore, we assume the independence of each dimension of the base distribution, as (Rudolph, Wandt, and Rosenhahn 2021;Yu et al. 2021; Gudovskiy, Ishizaka, and Kozuka 2022) did. i.e. Σ is assumed to be the diagonal matrix.
AltUB trains the value of parameters of the base distribution effectively, hence increasing the expressive power while helping to detect anomalies. Because AltUB is simple and independent of an upstream model, the proposed method can be applied to any normalizing flow model for anomaly detection.

Experimental Setups
Datasets We evaluate our method on two datasets: MVTec AD (Bergmann et al. 2019) and beanTech Anomaly Detection(BTAD) (Mishra et al. 2021a) which are well-known as benchmark of anomaly detection models.
MVTec AD is a widely utilized dataset specifically designed for unsupervised anomaly detection. It consists of 5354 high-resolution images of industrial objects or patterns. In detail, it is composed of a total of 15 categories with 3629 images for training and validation and 1725 images for testing. We compare the proposed method with SSP-CAB+DRAEM (Ristea et al. 2022;Zavrtanik, Kristan, and Skocaj 2021), PatchCore (Roth et al. 2021), and CS-Flow (Rudolph et al. 2022) which are current state-of-the-art models.
BTAD is a real-world dataset for industrial unsupervised anomaly detection. It comprises 2540 images divided into three categories of industrial products. The performance of our method on BTAD is compared with PatchSVDD (Yi and Yoon 2020), VT-ADL (Mishra et al. 2021b), and the method introduced in (Tsai, Wu, and Lai 2022).
Both MVTec AD and BTAD have only normal images in the training set whereas normal images and abnormal images are together in the test set.
Metrics The performance of models is measured by the Area Under the Receiver Operating Characteristic curve (AUROC) at the image or pixel level. which is a general metric for the evaluation of anomaly detection models. Meanwhile, the expected value of the performance is more practical than the best performance during training in unsupervised tasks because the validation set is not provided in the real world. In particular, the variance of the performance is also significant for stability. To achieve the purpose of realworld anomaly detection tasks, both the best and the average value of AUROC evaluated during the training process (from 100th epochs to 300th epochs) are adopted for comparison of normalizing flow-based models. We provide (mean, standard deviation) for average performance.

Anomaly detection and segmentation performance
We apply AltUB to FastFlow and CFLOW-AD which are well-known and well-performed NF models for anomaly detection. To be specific, we compare the performance of these original models and those of applied AltUB.  Figure 4: The pixel-wise AUROC is measured on the pill category of the MVTec AD dataset. FastFlow based on CaiT and that applied AltUB are compared.
FastFlow clearly achieves the new state-of-the-art performance on the anomaly segmentation task of the MVTec AD dataset. In addition, AltUB improves the stability of performance of NF models as Table 2, in terms of the point that AltUB enhances the average performance and reduces its variance in most cases. One sample is depicted in Figure  4. This observation is consistent with FastFlow based on the other feature extractors. The full information about the performance is provided in the supplementary material A1.
BTAD For generalization, we evaluate the proposed method with BTAD. As Table 3 and Table 4, AltUB also increases both the best performance and the stability of performance on BTAD except in one case. Our method also establishes the state-of-the-art performance on the anomaly segmentation task of BTAD.

Examination for normalization
To examine whether AltUB is effective to update the base distribution, a one-sample Kolmogorov-Smirnov test (KStest) is utilized. One-sample KS statistic for an empirical cumulative density function (cdf) F n and a given cdf F is defined as where n is the number of independent and identically distributed samples for the empirical distribution. KS(·, ·) converges to 0 as the two distribution is similar. Ideally, each spatial location of each channel of outputs corresponding to normal inputs shows the N (0, 1).
For  Table 5 apparently shows that our method achieves the lower KS statistic, Table 1: Official anomaly detection and segmentation performance on MVTec AD dataset with the format (detection AUROC, segmentation AUROC). The performance of the proposed method is recorded as the best one during the training process. For FastFlow, the best one is chosen between the four distinct feature extractors. The performance with bold letters means the best one among these models. meaning that the base distribution is trained effectively. This observation supports our hypothesis that targets the expressive power and reveals the reason why AltUB can achieve higher and more stable performance effectively.

Comparison with other methods to update the base distribution
To justify the necessity of alternating training, we compare AltUB with other methods to train the base distribution in Table 6. As we mentioned before, the general training (stereotype) with an original learning rate isn't effective as the learning rate isn't large enough. Update using CNN decreases the performance because CNN can't distinguish anomalies and the parameters of the base distribution are sensitive to the weight of CNN.

Inference time
Because AltUB doesn't require complex calculations, the inference time has hardly increased. The analysis results are shown in Table 7. The inference time is measured by Intel(R) CPU i7-12700K and NVIDIA GeForce RTX 3060.

Implementation details
In our method, ln Σ 1 2 is trained instead of Σ to have negative values for itself. In addition, µ and ln Σ 1 2 are initialized to zero not to disrupt the initial training of the flow model.
We set the freezing interval as 5 epochs and η 2 as a product of a constant and the original learning rate η 1 , The constant is determined that the maximum value of η 2 equals 0.05. Exceptionally AltUB is not applied during the warmup process of CFLOW-AD. We apply gradient clipping as much as 100 to the base distribution for FastFlow to prevent divergence of loss. Layer normalization is applied to outputs of FastFlow with WRN50 and CaiT. We fix the other hyperparameters of flow models as their original papers.

Conclusion
In this paper, we propose a simple yet effective method Al-tUB to enhance the expressive power of normalizing flow for anomaly detection. Our key idea is an alternating training method to learn the base distribution. In our experimental results, we prove the proposed method improves the stability of flow models for anomaly detection.
Limitations and future works Despite the success of our method, the performance still fluctuates in some cases. This fact implies that a more suitable training methodology can be suggested. From the point that our simple idea achieves meaningful progress for anomaly detection, an essential approach toward the higher expressive power to detect anomalies will be required.