{'fraction_non_alphanumeric': 0.031991356226805094, 'fraction_numerical': 0.011979142199464462, 'mean_word_length': 5.13840830449827, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
WaveNets: Wavelet Channel Attention Networks

Channel Attention reigns supreme as an effective technique in the field of computer vision. However, the proposed channel attention by SENet suffers from information loss in feature learning caused by the use of Global Average Pooling (GAP) to represent channels as scalars. Thus, designing effective channel attention mechanisms requires finding a solution to enhance features preservation in modeling channel inter-dependencies. In this work, we utilize Wavelet transform compression as a solution to the channel representation problem. We first test wavelet transform as an Auto-Encoder model equipped with conventional channel attention module. Next, we test wavelet transform as a standalone channel compression method. We prove that global average pooling is equivalent to the recursive approximate Haar wavelet transform. With this proof, we generalize channel attention using Wavelet compression and name it WaveNet. Implementation of our method can be embedded within existing channel attention methods with a couple of lines of code. We test our proposed method using ImageNet dataset for image classification task. Our method outperforms the baseline SENet, and achieves the state-of-the-art results. Our code implementation is publicly available at https://github.com/hady1011/WaveNet-C.


I. INTRODUCTION
In deep convolutional neural networks (CNNs), effective feature learning often relies upon the success of attention mechanisms in selectively capturing and preserving relevant important details from input [1]. In tasks such as image classification, attention mechanisms involve redistributing the weights of input feature maps to achieve better classification accuracy [2], [3]. Major attention mechanisms used in CNNs consist of channel attention, spatial attention, branch attention, and temporal attention. [4], [5]. Particularly, the computer vision domain conventionally adopts the channel attention (CA). Introduced by the SeNet [6], CA offers a relatively computationally efficient selection of important channels by generating scalar channel weights, whereby channel-wise computations are performed on features derived from the global average pooling (GAP).
While channel attention is an intuitive technique in capturing salient properties of images, recent studies suggest that CA's use of global average pooling (GAP) in its architecture hinders its performance. GAP is insufficient in retaining sophisticated details and fails to comply with some taskspecific model practices [7]. Moreover, GAP's straightforward dimensionality reduction further limits CA's inter-channel dependencies modeling [8]. Our motivation to design WaveNet stems from this need to reassess CA to capture finer details in feature learning. This reassessment should allow CA to redistribute the weights of input feature maps to improve classification accuracy while maintaining CA's computational efficiency.
To address the above limitations, we propose to enhance the feature preservation during downsampling via the discrete wavelet transform (DWT). As a tool in digital signal processing, DWT has various image processing applications in tasks such as image compression, dehazing, classification, denoising, restoration, and watermarking [9] [10] [11] [12] [13]. Essentially, DWT performs pyramidal image decomposition by transforming an image into four sub-bands composed of a lowpass (LL) filter and a bandpass filter with horizontal (LH), vertical (HL), and diagonal (HH) decomposition of the image, respectively [14] [15]. The LL filter corresponds to a downsized version of the original image with lower resolution, and the LH, HL, and HH bandpass filters highlights the input's predominant traits in their associated orientation. The ability of DWT to perform multilevel decomposition on images inspired WaveNet, in which we explore the levels of decomposition of DWT's application in CA.
In this work, we introduce a novel channel attention framework that stems from a mathematical compression technique. In an effort to better represent channel information and express what GAP failed to explore, we propose to utilize Haar DWT for the channel attention mechanism. Along with the Haar channel attention framework, we propose a customized wavelet channel attention framework. In this framework, we use a set of random orthogonal filters to be used in a customized wavelet. The role of those orthogonal filters is to enforce feature preservation and diversity in the compression task prior to excitation of channel attention.
Our implementation of this enhanced channel attention mechanism achieves the state-of-the-art performance against related channel attention techniques. The main contribution of this work are summarized as follows: • We view the channel attention from a compression perspective and adopt DWT in the vanilla channel attention for channel information preservation. With the proof, we establish that conventional GAP is the recurrent Approximate Component of Discrete Haar Wavelet Transform. Then, we generalize the channel attention from the frequency basis and propose our method, termed as WaveNet. • Motivated by the success of the Discrete Haar Wavelet Transform in WaveNet, we propose WaveNet-C, a custom orthogonal linearly independent filters wavelet to enforce diversity in the compression task for Channel Attention. • We propose a filter selection criteria along with a parameter reduction technique to fulfill WaveNet-C. • [27]. Early interest in visual attention is fostered by the highway network [28], which introduced a gating mechanism that enhances the flow of feature information in a deep neural network. ResNet's [29] success with deep CNNs via the use of skips connections in residual blocks further set the foundation for using attention in creating the next state-of-art model. Soon, the proposal of SENet [6] presents the channel attention in an efficient squeeze and excitation architecture, fueling a wave of studies aiming to improve the channel attention performance. Notably, DANet [30] integrates a position attention module with channel attention to model long-range contextual dependencies. Building upon NLNet [31] and SENet, GCNet [32] proposes GC blocks to capture channel-wise interdependencies while emphasizing long-range global context modeling. [33] introduces the triplet attention, modeling spatial attention and channel attention with efficient parameters and no dimensionality reduction. HA-CNN [34] assesses joint attention selection, which combines hard regional and soft spatial attention with channel attention. Besides utilizing spatial attention, CBAM [35] applies global maxpooling in channel attention to counter GAP's limits. ECA-Net [36] remodels the channel attention architecture to capture cross-channel interaction without unnecessary dimensionality reduction. TSE [37] disregards GAP's global spatial context in SENet to streamline the SE network usage with AI accelerators. FcaNet [38] adds a multi-spectral component to channel attention from a frequency analysis perspective that explains the relationship between GAP and discrete cosine transform. b) Wavelet Transforms in Image Processing: Wavelet transforms attract growing interest in deep learning-based image processing applications. Early associated works tend to neglect the use of attention mechanisms and range from image super-resolution [39], [40], classification [41] [42], inpainting [43], demoiréing [44], and restoration [12]. Recently, some works propose to integrate attention mechanisms and Wavelet transforms. AWNet [45] integrates non-local attention with DWT to achieve better results for image signal processor (ISP) with smartphone images. [46] proposes WAEN, which composes of an attention embedding network and a wavelet embedding network to enhance video super-resolution. The soft attention-based model proposed in [47] applies DWT to improve face recognition of morphed images. [48] presents a framework for detecting surface defects of glass bottles that fuses the Wavelet transform into their visual attention model. [49] details a single image deraining framework based on a fusion network with DWT and its inverse into its attention module. Different than most previous works that use wavelet transform with an attention mechanism for a specific domainbased application, our WaveNet seeks to incorporate Wavelet transform into the underlying architecture of CA to improve the attention mechanism at its most fundamental level.

III. METHOD
We start this section by formulating the Discrete Wavelet Transform (DWT) and Channel Attention (CA). We then look into more details over the derivation of our Interdependent channel attention. Together with the interdependent channel attention model, we explore a diversification strategy for custom wavelet transform.

A. Discrete Wavelet Transform (DWT) and Channel Attention (CA)
In this section, we first go in-depth over the mathematical derivation of DWT. Then, we elaborate on explaining the channel attention mechanism. a) DWT using Multiplication: Given scale weights H and shift weights G describing wavelet w, the wavelet transform of (1D) input X is: Where The (2D) DWT can be described using the (1D) DWT by applying the procedure to columns first then repeating the process to the rows of the output. The first level DWT for input X can be modeled as follows: Where A is the Approximation of X, V is the Vertical difference of X, H is the Horizontal difference of X, and D is the Diagonal difference of X. For an image with the size of (n × n), the extracted features size is (n/2 × n/2). The wavelet transform level is the number of the times of wavelet feature extraction. At J th level, the extracted feature size is (n/2 J × n/2 J ).
b) DWT using Convolution: Given decoding high pass and low pass filters H, L respectively, we use convolution with correlations of the form : We assemble the encoding filters by stacking the Low-Low, horizontal, vertical, diagonal filters. c) Channel Attention: Convolution Neural Networks rely heavily on channel attention mechanisms. The idea is to recalibrate the channel weights based on relative importance to the general task. Suppose that X ∈ R C×H×W is an instance of a deep image feature, C is the channel count, H is the feature height, and W is the feature width. As discussed in Sec. I, the channel attention process aims to summarize the channel content into a scalar value. Hence the channel attention mechanism described initially by SENet [] can be written as: where att ∈ R C is attention vector, sigmoid is Sigmoid function, f c is a mapping function such as a fully connected layer or an one-dimensional convolution, and squeeze (GAP): R C×H×W → R C×1×1 is a compression method. After acquiring the attention vector of all C channels, all channels of input X are scaled by their corresponding importance value: X N,C,W,H = att N,C,1,1 · X N,C,W,H where att, X, and X are the input, attention vector and attention mechanism output. Typically, global average pooling is used as the compression method [6], [36]. Other popular compression methods are global max pooling [35] and global standard deviation pooling [50].

B. Interdependent Channel Attention
In this section, we start by highlighting weaknesses of the current channel attention mechanisms. Based on the theoretical analysis, we then discuss our proposed design to overcome those weaknesses. a) Wavelet Channel Attention: As discussed in Sec. III-A, DWT extracts four main features of an image. With the proof, We demonstrate that GAP is equivalent to the recurrent approximation of the input image when Haar wavelet transform is used. Proof. The proof is divided into two transforms. The first transform is applied to the input image X to get a padded version with equivalent global average pooling. If the image X isn't divisible by 2 in both dimensions, we pad X to get an image A = P (X) with GAP (X) = GAP (A). If the input image is already divisible by 2 in both dimensions, we define P to be the identity function.
Next we get GAP (B) = GAP (P (A)) = GAP (X). You can repeat this argument until B is a 1x1 and GAP (B) = b. To do so, we introduce the second transform T. If B = T (A), for T being the transform, we have From this, it follows that which implies that GAP (B) = GAP (A). Since GAP (A) = a if a is a 1x1 matrix, the proof is complete by induction.

b) Orthogonal Linearly Independent Channel Attention
Module: Theoretical analysis and Theorem 1 support that GAP in the channel attention mechanism only uses the average approximation feature while a diverse variety of potential features are discarded. However, the discarded features may also encode the useful information patterns in representing the channels and should be taken into consideration in the compression phase. To mathematically derive a more diverse and meaningful compression method of channel information, we propose to generalize GAP to more wavelet filters and compress more information with multiple different wavelets.
ResNet has two main blocks, Basic Block and Bottleneck Block. Basic has 4 channel sizes 64, 128, 256, and 512. Bottleneck has 6 channel sizes 64, 128, 256, 512, 1024, and 2048. In case of basic Block, we initialize 4 random interdependent orthogonal filters of same size as channels and we train the network on those filters. For the Bottleneck, we use the same filters for the channels sizes that are shared with the Basic Block. For the channels 1024 and 2048, we split the channels to chunks of size 512 and we initialize those extra 4 filters of size 512 with new random orthogonal weights to enforce catching more diverse information during the compression phase.
The input X is passed through a separate orthogonal linearly independent wavelet compression module to represent diverse interdependent channel information. In this way, we express the basic compression (C B ) as follows: in which the recursive wavelet level J = log 2 {H}. X ∈ R C×H×W is the input feature, and C(X) ∈ R C is the Cdimensional vector post compression. Similarly, bottleneck compression (C BN ) is described as follows: C(X) BN = C B (X) C = 64, 128, 256, 512 CAT (C B (X 512 )) C = 1024, 2048 (11) where CAT is the concatenation function along the channel dimension and X 512 is the split of the input X of size 512 along the channel dimension.
The final orthogonal interdependent channel attention can be expressed as: From Eqs. 10, 11 and 12, it is demonistrated that out model performs a set of Wavelet transforms and extracts channel diverse compression representations of channel information. By incorporating those extra information in the final description we notice a major improvement in the channel representation. Fig. 1 illustrates the overall concept of our method.
c) Wavelet Filter Choice: One important decision for the network is to pick the wavelet to perform on a specific channel. Our baseline network named Wavenet perform Haar approximation on all channels and achieve SENet results. In order to fulfill the orthogonal interdependent channel attention, we propose Wavenet-C. We discuss more about those networks in the following subsections.
WaveNet means WaveNet weights the components of wavelet compression within each step of the deep wavelet compression. Its main idea is to improve the compression by including the vertical, horizontal, and diagonal components. First, the network determines the importance of each frequency component. Then, it investigates the effect of adding those frequency components together through the recurrence process.
WaveNet-C means WaveNet with selective wavelet filters. We use the convolution based wavelet transform and we assign orthogonal independent filters for channel compression. We do so by randomly initializing the filters then applying the gramschmidt process to orthogonality those filters thus forcing the network to diversify the information compressed by each channel therefore achieving better classification in general.

IV. EXPERIMENTS
In this section, we began by describing the experimental details of our implementation. Then, we discuss the technique of information compression in our framework, complexity, and code implementation. Lastly, we discuss the accuracy of our method on image classification, object detection, and instance segmentation tasks.

A. Implementation Details
We utilize ResNet-34, as backbone model to evaluate the proposed WaveNet on ImageNet [51]. We comply with data augmentation and hyper-parameter settings in [29] and [52]. Specifically, with random horizontal flipping, the input images are cropped randomly to 256×256. To do so, we modify ResNet architecture to allow the input size to be 256 instead of 224. During training, the SGD optimizer is set with a momentum of 0.9. The learning rate is 0.2, the weight decay is 1e-4, and the batch size is 256 per GPU. All models are trained within 100 epochs using Cosine Annealing Warm Restarts learning schedule and label smoothing. To foster convergence, for every 10 epochs, the learning rate scales by 10% of the previous learning rate. We further adopt the Nvidia APEX mixed precision training toolkit and Nvidia DALI library for fast data loaders for training efficiency.
All models are implemented in PyTorch [53] and tested on two Nvidia Quadro RTX 8000 GPUs.
B. Discussion a) How the Orthogonal Linearly Independent filters compresses and embeds more information: In Sec. III-B, we prove that solely adopting the vanilla GAP in the channel attention discards information from all filters except the Haar filter, i.e., GAP. Therefore, designing the filters to be orthogonal and linearly independent using the Gram-Schmidt method would force the network to diversify the information extracted in the channel attention compression phase.
We also provide a theoretical basis to show that more information could be embedded. By nature, deep networks are redundant [54], [55]. If two channels contain redundant information, then the application of GAP on these channels are likely to return repetitive information. On the other hand, our multi-spectral framework extracts less superfluous information from redundant channels since the inherent diverging frequency components contain different information. Thus, our multi-spectral framework can embed more unique salient information in the channel attention mechanism.
b) Complexity analysis: We analyze the complexity of our framework through the number of parameters and the computational cost. Our method does not impose no extra parameters compared with the baseline SENet that introduced the vanilla channel attention since the filters of 2D DWT are pre-computed constant. The negligible increase in the computational cost is also similar to computational cost of SENet. With ResNet-34 backbone, the relative computational cost increases of our method is 0.05% compared with SENet, respectively. More results can be found in Table I. c) A Few lines of code change: Another strength of the proposed wavelet attention framework is that it can be integrated into existing diverse variants of channel attention implementations. The major distinction between our method and SENet is the adoption of different channel compression method (multi-spectral 2D DWT vs. GAP). As discussed in Sec. III-A0b and Eq. 5, 2D DWT can be viewed as a constant filter convolution of inputs. It can be simply implemented via a Conv2D layer. Accordingly, arbitrary channel attention methods can adopt our framework easily.

V. CONCLUSION
In this paper, we proposed the WaveNet, an efficient, flexible framework for improving channel attention's power in capturing salient features that can easily incorporate into existing channel attention-based models. Theoretically, we prove that the conventional GAP is the recurrent approximation component of the DHWT that discards all channel information in all filters except the Haar filter. Hence WaveNet tackles channel attention as a compression problem and introduces DWT to preserve more unaccounted channel-wise features under GAP. We further introduce WaveNet-C, a custom orthogonal linearly independent wavelet to best fit the compression task for channel attention, and effective wavelet filter selection criteria and parameter reduction techniques. Empirically, our method persistently improves the performance of channel attention mechanism in ImageNet classification task without raising significant parameters and computation costs relative to existing frameworks. Our future works include extending our method for bigger ResNet networks like ResNet-50, ResNet-101; introducing other tasks and datasets like segmentation and object detection on COCO dataset; and incorporating delayed learning for the wavelet filters to further improve our method accuracy.
ACKNOWLEDGMENT This work was supported by the National Institute of General Medical Sciences of the National Institutes of Health under award P20GM139768, and the Arkansas Integrative Metabolic Research Center at the University of Arkansas. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.