{'fraction_non_alphanumeric': 0.04217774908543146, 'fraction_numerical': 0.01198457234609591, 'mean_word_length': 4.586978636826043, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Convolutional Framelets: A General Deep Learning for Inverse Problems

Recently, deep learning approaches have achieved significant performance improvement in various imaging problems. However, it is still unclear why these deep learning architectures work. Moreover, the link between the deep learning and the classical signal processing approaches such as wavelet, non-local processing, compressed sensing, etc, is still not well understood, which often makes signal processors in deep troubles. To address these issues, here we show that the long-searched-for missing link is the convolutional framelets for representing a signal by convolving local and non-local bases. The convolutional framelets was originally developed to generalize the recent theory of low-rank Hankel matrix approaches, and this paper significantly extends the idea to derive a deep neural network using multi-layer convolutional framelets with perfect reconstruction (PR) under rectified linear unit (ReLU). Our analysis also shows that the popular deep network components such as residual block, redundant filter channels, and concatenated ReLU (CReLU) indeed help to achieve the PR, while the pooling and unpooling layers should be augmented with multi-resolution convolutional framelets to achieve PR condition. This discovery reveals the limitations of many existing deep learning architectures for inverse problems, and leads us to propose a novel deep convolutional framelets neural network. Using numerical experiments with sparse view x-ray computed tomography (CT), we demonstrated that our deep convolution framelets network shows consistent improvement. This discovery suggests that the success of deep learning is not from a magical power of a black-box, but rather comes from the power of a novel signal representation using non-local basis combined with data-driven local basis, which is indeed a natural extension of classical signal processing theory.

regularizers, the authors in [15] tried to learn a set of optimal regularizers. Multilayer perceptron was developed for accelerated parallel MRI [32]. Domain adaptation from sparse view CT network to projection reconstruction MRI was also proposed [18]. These pioneering works consistently showed impressive reconstruction performances, which are often superior to the existing iterative approaches.
However, the more we have observed impressive empirical results in image reconstruction problems, the more unanswered questions we encounter. For example, to our best knowledge, we do not have the answers to the following questions that are critical to a network design: 1. What is the role of the nonlinearity such as rectified linear unit (ReLU) ? 2. What is the role of the filter channels in convolutional layers ? 3. Why do we need a pooling and unpooling in some architectures ? 4. Why do some networks need a fully connected layers whereas the others do not ? 5. What is the role of by-pass connection or residual network ? Furthermore, the most troubling issue for signal processing community is that the link to the classical signal processing theory is still not clearly understood. For example, wavelets [8] has been extensively investigated as an efficient signal representation theory for many image processing applications by exploiting energy compaction property of the expansion. Compressed sensing theory [10,5] has further extended the idea to demonstrate that an accurate recovery is possible from undersampled data, if the signal is sparse in some frames and the sensing matrix is incoherent. Non-local image processing techniques such as non-local means [4], BM3D [7], etc have also demonstrated impressive performance for many image processing applications. The link between these algorithms have been extensively studied for last few years using various mathematical tools from harmonic analysis, convex optimization, etc. However, recent years have witnessed that a blind application of deep learning toolboxes sometimes provides even better performance than mathematics-driven classical signal processing approaches. Does this imply the dark age of signal processing or a new opportunity ? Therefore, the main goal of this paper is to address these open questions. In fact, our paper is not the only attempt to address these issues. For example, the interpretation of a deep network in terms of unfolded (or unrolled) sparse recovery has been one of the prevailing views in research community [14,52,15,23]. However, this interpretation still does not give an answer to several key questions: for example, why do we need multichannel filters ? In this paper, we therefore depart from this existing view and propose a new interpretation of a deep network as a novel signal representation scheme. In fact, signal representation theory such as wavelets and frames have been active areas of researches in many years [36], and Mallat [37] and Bruna et al [3] recently proposed a wavelet scattering network as a translation invariant and deformation-robust image representation. However, this approach does not have learning components as in the existing deep learning networks.
Then, what is missing here? One of the most important contributions of our work is to show that the geometry of deep learning can be revealed by lifting a signal to a high dimensional space using Hankel structured matrix. More specifically, the lifted Hankel structure matrix can be factorized into non-local, and local basis matrices as well as a sparse matrix that has energy compaction property. This leads to a frame representation of signals using two bases: non-local and local bases. Our novel discovery is the realization that the non-local basis determines the network architecture such as pooling/unpooling, whereas the local basis enables the network to learn convolution filters. More specifically, for a given nonlocal basis, the local basis is trained such that it provides the maximal energy compaction of true signals while the noise and artifacts can be spread out. In other words, the application-specific knowledge leads to a better choice of a fixed non-local basis, on which the local basis are learned to maximize the performance.
We demonstrate that the power of deep learning indeed comes from the optimal interplay between the nonlocal and local bases. In fact, the idea of exploiting the two bases by the convolutional framelets was originally proposed by Yin et al [54]. However, the aforementioned close link to the deep neural network was not revealed in [54]. Most importantly, we demonstrate for the first time the importance of the perfect reconstruction (PR) condition under rectified linear unit (ReLU). Furthermore, the mysterious role of the redundant multichannel filters and residual network can be easily understood as important tools to satisfy the PR condition under ReLU. In particular, by augmenting local filters with paired filters with opposite phase, the ReLU nonlinearity disappears and the deep convolutional framelet becomes a linear signal representation. In addition, to overcome the limitation of the pooling and unpooling layers, we derive a multi-resolution analysis (MRA) for convolutional framelets using wavelet-based non-local basis. However, to retain the deep network linear satisfying PR, the number of channels should increase exponentially along the layer, which is not practical. Thus, the number of filter channel are not usually sufficient at high layers, which results in a low-rank approximation of Hankel matrix at those layers. In fact, this interplay between the PR and low-rank Hankel matrix approximation is believed to the origin of the superior performance of deep learning.
Seeing with the new eyes of deep convolutional framelets, we analyze the limitations of the existing deep learning architecture for inverse problems. Accordingly, we propose a new class of deep learning network which results in consistent performance improvement over the existing deep learning approaches. We call the new class of deep network using convolutional framelets as the deep convolutional framelets.
1.1. Notations. For a matrix A, R(A) denotes the range space of A and N (A) refers to the null space of A. P R(A) denotes the projection to the range space of A, whereas P ⊥ R(A) denotes the projection to the orthogonal complement of R(A). The notation 0 p×q means the p × q zero matrix. The n × n identity matrix is referred to as I n×n . For a given matrix A ∈ R m×n , the notation A † refers to the generalized inverse. The superscript of A denotes the Hermitian transpose. Because we are mainly interested in real valued cases, is equivalent to the transpose T . The inner product in matrix space is defined by A, B = Tr(A B), where A, B ∈ R n×m . For a matrix A, A F denotes its Frobenius norm. For a given matrix C ∈ R n×m , c j denotes its j-th column, and c ij is the (i, j) elements of C. If a matrix Ψ ∈ R pd×q is partitioned as Ψ = Ψ 1 · · · Ψ p with sub-matrix Ψ i ∈ R d×q , then ψ i j refers to the j-th column of Ψ i .

Low-Rank Hankel Matrix Approaches.
In this section, we review the theory of annihilating filter-based low-rank Hankel matrix approach (ALOHA) [53,25,22,41,34,35,24], which was recently proposed as powerful image processing and inverse problem techniques. These turn out to be the key ingredients of the proposed method. For simplicity, our theory is developed using 1-D signal model, but the extension to multi-dimensional signal model is straightforward. We first begin with some mathematical preliminaries.

Hankel Matrix.
A (wrap-around) Hankel matrix can be easily obtained from (circular) convolution [53]. In this paper, to avoid special treatment of boundary condition, our theory is mainly derived using circular convolution. More specifically, let f = [f [1], · · · , f [n]] T ∈ R n and ψ = [ψ [1], · · · , ψ[d]] T ∈ R d be an input image and a convolutional filter kernel. Then, a single-input single-output (SISO) convolution can be represented in a matrix form: where Similarly, a single-input multi-ouput (SIMO) convolution using length-d filters ψ 1 , · · · , ψ q ∈ R d can be represented by where Y := y 1 · · · y q ∈ R n×q , Ψ := ψ 1 · · · ψ q ∈ R d×q , and q denotes the number of output channels. On the other hand, multi-input multi-output (MIMO) convolution can be represented by where p and q are the number of input and output channels, respectively; ψ j i ∈ R d , i = 1, · · · , q; j = 1, · · · , p refer to the corresponding filters with the filter length d. The corresponding matrix representation is then given by where F := [f 1 · · · f p ] and H d|p (F ) is an extended Hankel matrix by stacking p Hankel matrices side by side: and For notational simplicity, we denote H d|1 ([f ]) = H d (f ). Note that the aforementioned matrix vector operations are equivalent to the filtering operation used in the existing convolutional neural networks (CNN) [31] as shown in Fig. 1(a)-(c).
We denote the space of the wrap-around Hankel structure matrices of the form in (2) as H(n, d) and an extended Hankel matrix composed of p Hankel matrices of the form in (6) as H(n, d; p). The following elementary algebra in H(n, d) and H(n, d; p) (some are obtained from [53] and [54]) are useful.
Lemma 1. Let the space of wrap-around Hankel matrix H(n, d) be equipped with the matrix inner product A, B = Tr(A B). For k = 1, · · · , n, define where e k denotes the standard coordinate vector in R n with only the k-th element as one. Then, the following statements are true.
1. The set {A k } n k=1 is the orthonormal basis for the space of H(n, d).

2.
For a given f ∈ R n and the associated Hankel matrix H d (f ) ∈ H(n, d), we have 3. Let u ∈ R n and v ∈ R d . Then, for any Hankel matrix where denotes the circular convolution. 4. For a given A k in (7) and any u ∈ R n and v ∈ R d ,

A generalized inverse of the lifting (8) is given by
where B is any matrix in R n×d . 6. ForΦ ∈ R n×n and C ∈ R n×q andΨ ∈ R d×q , where c j denotes the j-th column of C and c ij is the (i, j) elements of C. 7. Let H † d|p (·) denote the generalized inverse of an extended Hankel operator H d|p (·). Suppose, furthermore, C = C 1 · · · C p ∈ R n×dp with C i ∈ R n×d , i = 1, · · · , p. Then, we have 8. For f ∈ R n and Ψ,Ψ ∈ R d×q , 9. For [f 1 , · · · , f p ] ∈ R n×p and Ψ,Ψ ∈ R pd×q with where ψ j i (resp.ψ j i ) denotes the i-th column of Ψ j (resp.Ψ j ). Proof. See Appendix.

Low-rank Hankel Structured Matrix Approaches for Inverse Problems.
Hankel matrices arise repeatedly from many different contexts in signal processing and control theory. In control theory, the low rank Hankel matrix has been used in system identification [12]. In signal processing theory, Hankel matrix has been widely used for harmonic retrieval and array signal processing [21], subspace-based channel identification [48], just to name a few. Recent advance in this field [53] is the discovery that a low-rank Hankel matrix is closely linked to the sampling theory of signals with the finite rate of innovations (FRI) [50]. More specifically, the rank of the Hankel matrix is directly related to the length of the annihilating filter as shown in the following theorem: [53] Let r +1 denote the minimum length of annihilating filters that annihilates data f = [f [1], · · · , f [n]] T . Then, for a given Hankel structured matrix H d (f ) ∈ H(n, d) whose number of columns and rows are bigger than r, we have where rank(·) denotes a matrix rank.
Here, the annihilating filter ψ, which is the key component of FRI sampling theory [50], is defined as the filter satisfying Vetterli et al [50] derived the explicit forms of the minimum length annihilating filters for FRI signals, which shows that the minimum annihilating filter length is directly related to the rate of innovation or the sparsity level of the original signal in the reciprocal domain [50]. Therefore, the rank of the Hankel matrix is determined by the sparsity level of the signal in the reciprocal domain [53]. Furthermore, we provided the necessary and sufficient condition for a Hankel matrix to have low rank property, which showed that a generalized form of FRI signals with (damped) exponentials is the only signal class that has low rank Hankel matrix in the reciprocal domain [53].
In [22], we also showed that the rank of the extended Hankel matrix in (6) can be further reduced when the multiple signals f 1 , · · · f p share the same support in the reciprocal domain or have the joint sparsity [22]. More specifically, if there exists inter-channel annihilating filter {h j } such that then the extended Hankel matrix H d|p (F ) have additional null space which makes it more low-ranked [22]. The existence of inter-coil annihilating filter is closely related to the joint sparsity [22].
The low-rank Hankel matrix is very useful in many inverse problem as demonstrated by many applications [25,22,41,34,35,24]. For example, if the Fourier data is sparsely sampled, the missing Fourier data can be interpolated by lifting the Fourier data into a Hankel structure matrix and applying the low-rank matrix completion algorithms [22,41,34,35]. In these applications, the lowrank Hankel matrix is constructed in the Fourier domain; however, the dual construction of low-rank Hankel matrix in the image domain is possible. More specifically, in [25] we showed that when an image is partitioned into smooth, edged or textured patches, the corresponding spectrum is sparse; thus, the lifted Hankel matrix can be constructed in the image domain to reveal the low rank structure. This idea can be used for image denoising [26] and deconvolution [39] by modeling the underlying intact signals to have low-rank Hankel structure, from which the artifacts or blur components can be easily removed. Moreover, the joint sparsity was exploited using extended Hankel matrix for parallel MRI [22], MR echo planar imaging (EPI) ghost artifact correction [35], etc. We will show later that the low-rankness of extended Hankel matrix is important for the perfect reconstruction condition for deep convolutional framelet expansion. However, the main disadvantages of the existing low rank Hankel matrix approaches [25,22,41,34,35,24] is the computational complexity due to the matrix factorization in each iterative step.
Note that the low-rank Hankel matrix algorithms usually assume the form of patch-by-patch processing in the low level computer vision problems such as denosing [26] or inpainting [25], whereas the global k-space data processing is used for MR applications [22,41,34,35]. It is also interesting to note that this is similar to the current practice of deep network for image reconstruction and low level computer vision applications, where the image domain approach is usually done as a patch-based processing, while the whole k-space processing is often used for deep learning-based MR reconstruction [15]. Later, we will show that this is not a coincident; rather it suggests an important connection of low-rank Hankel matrix approach to deep learning.

Convolutional Framelets and Its Extension.
In this section, we review the theory of convolutional framelets [54] exploiting the low rank Hankel matrix structure, and then discuss our novel extensions.
3.1. Review. The convolutional framelets by Yin et al [54] is a novel signal representation scheme combining the local and nonlocal characterization of patches in an image. The representation scheme was shown to be equivalent to a tight frame constructed from convolving local basis with nonlocal basis. Specifically, an image patch is first lifted to a form of Hankel matrix, from which nonlocal and local bases are obtained. This results in sparse expansion coefficients if the underlying Hankel matrix has low dimensional structure [54].
Specifically, for a given n-dimensional vector f ∈ R n , let the image patch lifted to Hankel structured matrix be denoted by H d (f ) ∈ H(n, d). Suppose, furthermore, that Φ and Ψ denote the two orthonormal matrices of dimension n × n and d × d, respectively, where φ i and ψ j refer to the columns of Φ and Ψ, respectively. Then, Yin et al showed the following result [54]: ). Let φ i and ψ j denotes the i-th and j-th columns of orthonormal matrix Φ ∈ R n×n and Ψ ∈ R d×d , respectively. Then, for any n-dimensional vector f ∈ R n , we have Furthermore, φ i ψ j with i = 1, · · · , n; j = 1, · · · , d form a tight frame for R n with the frame constant d.
Yin et al argued that the convolution frame φ i ψ j has both local and non-local properties, claiming that {ψ j } d j=1 forms either local or non-local basis whereas {φ i } n i=1 forms the other [54]. They further maintained that the superior energy compaction property emerges when the two bases are chosen to exploit the low-dimensional structure of Hankel matrix [54]. By energy compaction, it is implied that there are only a few non-zero values for the expansion coefficients or at least most of the energy is concentrated in only a few expansion coefficients. Specifically, Yin et al [54] solves the following optimization problem for a given orthonormal basis Φ: which maximizes the elements in the upper triangular part of upper r × r block. Thus, the design criterion by Yin et al [54] attempts to maximize energy concentration in this region, and the degree of energy concentration is again determined by the rank r of Hankel matrix.

Extensions.
While the original convolutional framelets by Yin et al [54] exploits the advantages of the low rank Hankel matrix approaches using two bases, there are several limitations.
Specifically, their convolutional framework uses only orthonormal basis. Second, the energy compaction property of Yin et al [54] to the upper triangular matrix is an optimization goal and it is not clear whether such energy compaction exists with the guarantee of perfect signal recovery. More importantly, the significance of multi-layer implementation using convolutional framelets was not noticed. In this section, we will address these limitations using the properties of Hankel matrix. This will be a basic building step to show that the cascaded convolutional framelets are directly related to a deep neural network when it is combined with nonlinearity.
Proposition 4. Let φ i and ψ j denotes the i-th and the j-th columns of matrix Φ ∈ R n×m and Ψ ∈ R d×q , respectively. Then, for any n-dimensional vector f ∈ R n , under the following resolution of we have or equivalently, where c j is the j-th column of the framelet coefficient matrix Proof. Using the resolution of identity (21) and (22), we have where C ∈ R m×q denotes the framelet coefficient matrix computed by and its (i, j)-th element is given by where we use (9) for the last equality. Furthermore, using (12) and (13), we have This concludes the proof.
Remark 1. In this paper, we call the column of the local basis Ψ andΨ as local filters and its dual filters. In addition, Φ is called as non-local transform because it transform the filtered signals by multipling Φ . This is because the local filter is usually constructed using small size convolutional filters, after which non-local transforms are applied to further decorrelate the remaining non-local correlation. If the nonlocal basis Φ is orthonormal, then Φ corresponds to the inverse non-local transform.
Remark 2. We can make the resolution of identity in (21) and (22) hold even if m > n or q > d. This means that the framelet expansion of non-local and local transform can be redundant.
Note that the so-called perfect recovery condition (PR) represented by (23) can be equivalently studied using: In general, for a given matrix input Z ∈ R n×p , the perfect reconstruction condition for a matrix input Z can be given by which is explicitely represented in the following corollary: Corollary 5. Let Φ,Φ ∈ R n×m and Ψ,Ψ ∈ R pd×q satisfy the resolution of identity in (21) and (22), respectively. Suppose, furthermore, that whose j-th column is represented by ψ i j andψ i j , respectively. Then, for any matrix Z ∈ R n×p , we have or equivalently, where c j is the j-th column of the framelet coefficient matrix Proof. For a given Z ∈ R n×p , using the resolution of identity (21) and (22), we have where C ∈ R m×q denotes the framelet coefficient matrix computed by and its (i, j)-th element is given by Furthermore, using (12), (13) and (14), we have This concludes the proof.

Examples.
3.3.1. Pooling and Unpooling. So far, we have claimed that the non-local basis Φ ∈ R n×m and its dualΦ should satisfy (21) for the PR condition. However, there are some popular non-local basis structures that do not satisfy the requirement. For example, pooling and unpooling layers in the deep neural network corresponds to a non-local transform that convert signals to a different resolution. These are quite often used in classifier design as well as segmentation network [40]. Specifically, the average and max pooling operators Φ ave , Φ max ∈ R n× n 2 for f ∈ R n is defined as follows: where {b i } i in max pooling are random (0, 1) binary numbers that are determined by the signal statistics. We can easily see that the columns of max pooling or average pooling are orthonogonal to each other; however, it does not constitute a basis because it does not span R n . Then, what does this network perform? Recall that the standard unpooling layerΦ has the same form of the pooling, i.e.Φ = Φ. In this case, under the biorthogonality condition for the local filters, i.e. ΨΨ = I d×d , the signal after pooling and unpooling becomes:f which is basically a low-pass filtered signal. This works fine for the segmentation problem; but, a care needs to be taken for the case of image restoration, because the goal is to keep the details of the signal. In fact, this limitation of pooling and unpooling can be easily addressed by the multi-resolution analysis of convolutional framelets, which will be discussed in detail later.

Spectral Basis.
Unlike the pooling and unpooling, the spectral basis still guarantees the perfect reconstruction even though the corresponding local or non-local transforms perform dimension reduction. More specifically, for a given input matrix C ∈ R n×p , suppose that the Hankel matrix H d|p (C) with rank r < pd have the following singular value decomposition: where U ∈ R n×r and V ∈ R pd×r denote the left and right singular vector bases, and Σ ∈ R r×r is the diagonal matrix whose diagonal components contains the singular values. Then, we can easily show that Proposition 4 can still hold with the nonlocal and local basis Φ,Φ ∈ R n×m and Ψ,Ψ ∈ R pd×q with m < n and q < pd satisfyingΦ This is because However, the correspondingΦ is usually a dense matrix dependent on the specific input signal. A data-dependent dense structure of unpooling matrix is prohibited due to its huge memory requirement. For example, if one is interested in processing 512 × 512 (i.e. n = 2 9 × 2 9 ) image using dense form of pooling/unpooling matrices, the required memory becomes 2 37 , which is not possible to store or estimate.
On the other hand, the local basis and its dual Ψ,Ψ can be learned to satisfy (33), because the dimension of the local basis is much smaller than the input signals. This fact is important to achieve PR condition under rectified linear unit (ReLU) using small set of convolutional filters. This issue will be revisited later.

Perfect Reconstruction and Inverse
Problems. So far, we have investigated the perfect recovery (PR) condition for convolutional framelets and we are now ready to explain why the perfect recovery condition is important for inverse problems. This is because the PR is usually associated with the energy compact property [36].
Consider an inverse problem that aims to obtain a noiseless signal f ∈ R n from a noisy measurement g ∈ R n through a sensing matrix A: where e ∈ R n is the noise. Suppose that we have an nonlocal transform Φ (0) that approximates a generalized inverse of the sensing matrix. Then, the application of the nonlocal transform leads to where w is considered as an artifact component. Because the noise is additive, if the measurement is lifted to a Hankel structured matrix, we have where Y, F, W ∈ H(n, d) are lifted wrap-around Hankel matrices from y, f and w, respectively. Now, the lifted Hankel matrix reveals important geometry. More specifically, F matrix from the noiseless signal is low-ranked as shown by the theory of low-rank Hankel matrix approach [53], whereas W is usually full-ranked because the noise does not have any low dimensional structure. Indeed, this observation has been successfully used in the annihilating filter-based low rank Hankel matrix approaches for image denoising [26], artifact removal [24] and deconvolution [39].
Thus, if we choose the nonlocal and local bases to have energy compaction, the most dominant convolutional framelet coefficients are from the underlying intact signals, so we can apply some operations to remove the remaining framelet coefficients and to obtain the noise reduced image. In the classical signal representation approaches such as wavelets, this is usually done using shrinkage operation such as soft-thresholding [9]. However, in the following section, we will show that there is a better way, which is directly related to neural network learning.

Deep Convolutional Framelets Neural Networks.
In this section, which is our main theoretical contribution, we will show that the convolutional framelets is directly related to the deep neural network when combined with multilayer implementation and nonlinearity. Moreover, our convolutional framelets can explain additional components of deep learning: redundant convolutional filters and advanced network components such as residual block [20]. Amazingly, the ReLU nonlinearity plays the key important role in this analysis.

From Convolutional Framelets to Neural Network.
In this section, we first show that the convolutional framelet expansion for a matrix input C ∈ R n×p can be written as convolutional layers. More specifically, (29) corresponds to a decoder layer: where the decoder-layer convolutional filter τ (Ψ) is defined by Now, the encoder layer computes the framelet coefficient matrix C using the following convolution: Note that there exists a major difference in the encoder and decoder layer convolution. Aside from the difference in the specific convolutional filters, note that the nolocal transform Φ should be applied later to the filtered signal in the case of encoder (36), whereas the nonlocal transform is applied first before the local filtering is applied in the decoder layer. This is because a non-local transform is used to remove the non-local correlation that are still remaining in the data-driven local filtering.
The resulting encoder and decoder layers are illustrated by the red and blue lines, respectively, in Fig. 2(a). This suggests that the convolutional framelets is closely related to deep learning if multiple convolutional framelet expansions are cascaded sequentially while retaining the encoderdecoder architecture (see Fig. 2(b)). Specifically, at the first layer, (23) can be represented by where the subscript and superscript index (l) represents the l-th layer. Similarly, we have Thus, the two layer implementation is given by In general, the l-layer implementation of the convolutional framelets is given by where d (i) and p (i) denotes the filter length and the number of input channels at the i-th layer, respectively. Note that the number of input channels should increase exponentially along the layers to satisfy the PR condition: Proposition 6. To achieve the PR condition, the number of Hankel blocks in the extended Hankel matrix in l-th layer or the number of signal to be filtered, should satisfy: , with p (0) = 1 = d (0) and d (i) is the filter length at the i-th layer. Accordingly, the minimum number of required filter channel at the l-th layer is given by Proof. We prove this by mathematical induction. At l = 1, the input signal is f ∈ R n , so we need Φ (1) H d (1) (f )Ψ (1) to obtain the filtered signal C (1) . Since H d (1) (1) and p (1) = 1, this generates p (2) = d (1) filtered output. At the l-th layer, we assume that (38) is true, which means that there are p (l) signals that needs to be filtered. Then, the filtering operation can be represented by (5) . Thus, to guarantee the PR, the dimension of Ψ (l) should be p (l) d (l) × p (l) d (l) . Therefore, this results in the the number of Hankel blocks in the (l + 1)-th layer as p (l+1) = p (l) d (l) . Q.E.D.
In order to reveal a more closer link between the convolutional framelets and the deep network, we further need to consider a nonlinearity. Note that the ReLU nonlinearity [13,19] is currently most widely used for deep learning approaches. Specifically, the ReLU ρ(·) is an element-wise operation for a matrix such that for a matrix A = [a ij ] n,m i,j=1 ∈ R n×m , the ReLU operator provides non-negative part, i.e. ρ(A) = [max(0, a ij )] n,m i,j=1 . Then, Eq. (37) can be equivalently represented using an encoderdecoder convolutional layer form: with the encoder layer Then, the training problem with ReLU is defined as follows: denote the input and target sample pairs. Then, the deep convolutional framelets training problem is given by where g is defined by (39) with ReLU for each layer.
Note that this is very similar to the existing CNN training. However, there exists fundamental differences. First, even with the ReLU nonlinearity, the next section shows that the PR is still guaranteed if sufficient number of filter channels are available. In this case, if the target sample y i is the same as the input f i , the training problem is nothing but the filter bank design problem to guarantee the PR. Note that this is not usually the case in the standard deep learning approaches, since they do not usually satisfy the biorthogonal conditions (21) and (22). Therefore, in order to differentiate our approach from the standard deep learning, we call the new network as deep convolutional framelets.

Role of Nonlinearity.
Recall that the PR condition for the deep convolutional framelets was derived without assuming any nonlinearity. Thus, the introduction of ReLU appears counterintuitive in the context of PR. Amazingly, in spite of ReLU nonlinearity, the following theorem shows that the PR condition can be satisfied when redundant filter channels are available. Here, by redundant filter channels, we refer the case with Ψ ∈ R d×2m where m ≥ d.
which guarantees the PR, i.e.
Proof. We will prove by construction. Let Accordingly, by choosing (42), we have where we use (44) for the third inequality. By applying this from l = L to 1 in (39), we can see that Remark 3. Eq. (42) in Theorem 8 predicts the existence of filter pairs with opposite phase. Amazingly, this theoretical prediction coincides with the empirical observation in deep learning literature. For example, Shang et al [44] observed an intriguing property that the filters in the lower layers form pairs (i.e., filters with opposite phase). To exploit this property for further network performance improvement, the authors proposed so called concatenated ReLU (CReLU) to explicitly retrieve the negative part of Φ F Ψ + using ρ Φ F (−Ψ + ) [44].
Remark 4. Note that (42) informs that there are infininte number of global mimizers for the minimization problem (41). In fact, the most important requirement for PR is the existence of opposite phase filters as in (42) rather than the specific filter coefficients. This may suggest the excellent generalization performance of a deep network even from small set of training data set.
Theorem 8 deals with the PR condition using redundant local filters Ψ ∈ R pd×2m with m ≥ pd. This can be easily satisfied at the lower layers of the deep convolutional framelets; however, the number of filter channels for PR grows exponentially according to layers as shown in (38). Thus, at higher layers of deep convolutional framelets, the condition m ≥ pd for Theorem 8 may not be satisfied. However, even in this case, we can still achieve PR as long as the extended Hankel matrix at that layer is sufficiently low-ranked.
Theorem 9 (PR under ReLU with Insufficient Filter Channels). For a given input C ∈ R n×p , let H d|p (C) denotes its extended Hankel matrix whose rank is r. Suppose, furthermore, thatΦΦ = I n×n . Then, there exists local filter and its dual Ψ ∈ R pd×2m and dual basisΨ ∈ R pd×2m with r ≤ m < pd such that C = H † d|p (Φρ Φ H d|p (C)Ψ Ψ ). Proof. We will prove by construction. From the proof of Theorem 8, we know that where the last equality comes from the orthonormality of Φ. Since m ≥ r, there always exist local basis and its dual Ψ + ,Ψ + ∈ R pd×m such that Ψ +Ψ+ = P R(V ) where V ∈ R pd×r denotes the right singular vectors of H d|p (C). Thus, By applying H † d|p to both side, we can conclude the proof. We believe that Theorem 9 is very important, because it shows the origin of the power of deep learning in the context of inverse problem. More specifically, the lower layers perform signal decomposition using various filter bank satisfying the PR conditions, after which high layers preserves PR only for the signals whose extended Hankel matrix is sufficiently low-ranked. We conjecture that the deep network is trained such that the convolutional framelets expansion is optimally fitted to the desired input/output relationship. The following toy example clearly shows the effect of low-rank approximation from insufficient channel in the context of deep convolutional framelets. Example 1. Consider a simple 2-level deep convolution framelets withΦ (i) = Φ (i) = I n×n : Suppose, furthermore, that Ψ (i) ,Ψ (i) ∈ R d (i) ×2m (i) satisfy (42) for i = 1, 2. Now, consider the following three scenarios: 1) Ψ (i) ,Ψ (i) satisfy (43) for all i; 2) only Ψ (2) ,Ψ (2) satisfy (43) due to the insufficient channels at the filter layer; 3) only Ψ (1) ,Ψ (1) satisfy (43) due to the insufficient channels at the second layer. For each scenario, the network outputf exhibits very different behaviour as shown in the following: 1. In the first case, (45) can be simplified tô where we use the fact Ψ (2) ,Ψ (2) satisfy (42) and (43). Now, using the fact that Ψ (1) andΨ (1) also satisfy (42) and (43), we havê Thus, the PR is guaranteed. 2. In the second case, Ψ (2) ,Ψ (2) satisfy (42) and (43), so we have (46). Now, using the fact that Ψ (1) andΨ (1) also satisfy (42), we havê denotes the i-th column of Ψ + and we use (15). 3. In the third case, Ψ (2) andΨ (2) satisfy (42) but not the condition (43). Then (45) can be simplified tof where Consider, furthermore, the following decomposition of Ψ

· · ·Ψ
(2) +,p (2) , Then, using (12) and (14), we have i where ψ +,(2) i andψ +,j,(2) i denote the j-column of Ψ (2) + andΨ (2) +,j matrices, respectively. Then, by using another application of (12) and p (2) = d (1) , we havê denote the j-column ofΨ (2) +,j matrix. Therefore, we havê Note that this is a complicated nonlinear combination of various filters and this combinatorial filter combination occurs even the case all the layers satisfies the PR except the highest layer. This is believed to generate the rich representation power of a deep network.
The low-rank approximation of Hankel matrix is good for reducing the noises and artifacts as demonstrated in image denoising [26], artifact removal [24] and deconvolution [39]. However, as the network gets deeper, more layers cannot satisfy the condition (38). Thus, successive applications of the CNN can also reduce the important signals. To address this, the residual net [20] is useful. Recall that the residual net (ResNet) has been widely used for image classification as well as image reconstruction. More specifically, the residual net architecture shown in Fig. 3 can be represented by where F := H d|p (C). The following result shows that the residual network works to truncate the least significant subspaces: Theorem 10 (PR under ReLU using Residual Nets). For a given input C ∈ R n×p , let H d p (C) denotes its extended Hankel matrix and its SVD is given by where u i and v i denotes the left and right singular vectors, and σ 1 ≥ σ 2 ≥ · · · ≥ σ r > 0 are the singular values and r denotes the rank. Suppose, furthermore, thatΦΦ = I n×n . Then, there exists local filter and its dual Ψ ∈ R pd×2m and dual basisΨ ∈ R pd×2m with r < m < pd such that where R(F ; Φ, Ψ,Ψ) is given by (47).
Proof. Again the proof is by construction. From the proof of Theorem 8, we know that Thus, Thus, if we choose Ψ + such that V Ψ + = 0, then H d|p (C) = ΦR(F ; Φ, Ψ + ,Ψ + ) and we can guarantee (49). Now, the remaining issue to prove is the existence of Ψ + such that V Ψ + = 0. This is always possible if r < m. This concludes the proof.
Note that the PR condition for the residual network is much more relaxed than that of Theorem 9. Specifically, for the case of Theorem 9, the number of channels for the positive polarity filters should be at least same as the rank of the extended Hankel matrix, which may be still large enough. On the other hand, the PR condition for residual net only requires the existence of the null space in the extended Hankel matrix, which is relatively easy to meet. We conjecture that this may be one of the reason that residual net results in significant improvements.

Case Study.
In the following, we discuss special cases of deep convolutional framelets, which correspond to the specific examples of existing deep learning architectures for inverse problems.

Wavelet Residual Network (WavResNet).
By extending [27], Kang et al [29,28] proposed a wavelet domain residual learning (WavResNet) for low-dose CT reconstruction combined with multiple residual blocks as shown in Fig. 4. The key idea of WavResNet is to apply the directional wavelet transform first, after which a neural network is trained such that it can learn the mapping between noisy input wavelet coefficients and noiseless ones [29,28]. In essence, this can be interpreted as a deep convolutional framelets with the nonlocal transform being performed first. The remaining layers are then composed of CNN with local filters and residual blocks. Thanks to the global transform using directional wavelets, the signal becomes sparse, which is the main source of the advantages compared to the simple CNN. Another uniqueness of the WavResNet is the concatenation layer at the ends that performs additional filters by using all the intermediate results. This layer performs a signal boosting [29]. However, due to the lack of pooling layers, the receptive field size is smaller than that of U-net as shown in Fig. 5. Accordingly, the architecture was better suited for localized artifacts from low-dose CT noise, but it is not effective for removing globalized artifact patterns from sparse view CT. Fig. 4. WavResNet architecture for low-dose CT reconstruction [29,28].  [57] is a recently proposed neural network approach for image reconstruction, which is claimed to be general for various imaging modalities such as MRI, CT, etc. A typical architecture is given in Fig. 6. This architecture is similar to Fig. 4, except that the first layer nolocal transform Φ (1) is learned as a fully connected layer. Moreover, the original signal domain is the measurement domain, so only local filters are followed in successive layer without additional fully connected layer for inversion. In theory, if Φ (1) is learned to make Φ (1) H d (f ) to be sufficiently sparse, then its is believed to be the advantageous over standard CNN.

AUTOMAP. AUtomated TransfOrm by Manifold APproximation (AUTOMAP)
However, in order to use the fully connected layer as the nonlocal transform, a huge size network is required. For example, as shown in Fig. 6, in order to recover N × N image, the number of parameters for the fully connected layer is 2N 2 ×N 2 (see [57] for more calculation of required parameter numbers). Thus, if one attempts to learn the CT image of size 512 × 512 (i.e. N = 2 9 ) using AUTOMAP, the required memory becomes 2N 4 = 2 37 , which is neither possible to store nor to avoid any overfitting during the learning. This is another reason we propose to use analytic nonlocal transform such as wavelets in the next section and the companion paper [29]. Specifically, this is because wavelet transforms do not need to be stored or learned but still provides near optimal nonlocal transform for piecewise smooth functions [8].

U-Net architecture.
The U-net is composed of encoder and decoder network with skipped connection. The encoder and decoder network utilizes the pooling and unpooling as shown in Fig. 7 to utilize the exponentially large receptive field (see Fig. 5). As discussed before, the pooling and unpooling layers corresponds to the low-pass filtering of the input signals and the application of the short length filters further approximates the signal using low-rank approximation. On the other hand, the skipped connection works as residual network, so the net effect is to truncate the least significant signals. Therefore, this network works very well with estimating globalized artifacts such as streaking artifacts, etc [23,17]. However, the detailed information is often lost during pooling, so it is not as effective as WavResNet in removing localized noises.

Multi-Resolution Deep Convolutional Framelets for Inverse
Problems. In deep convolutional framelets, the non-local basis Φ is one of the important design parameters that controls the overall structure. In particular, the energy compaction property for the deep convolutional framelets is significantly affected by Φ, because it performs additional non-local transforms for the filtered signal using data-driven local basis. Recall that a spectral basis for the Hankel matrix results in the best energy compaction property. However, it is a data-dependent basis whose size is usually very large, so it is difficult to store and estimate. Moreover, depending on the data, the spectral basis varies so that we cannot use the same optimal spectral basis for various input data. Therefore, we should choose an analytic non-local basis Φ such that it can approximate the spectral basis and result in good energy compaction property. In this sense, wavelet is one of the best choices since it is known to be near optimal for piecewise continuous signals and images. Here, we still use the standard pooling and unpooling networks as low-frequency path of wavelet transform, but there exists additional high-frequency paths from wavelet transform. Another important motivation for multi-resolution analysis of convolutional framelets is the exponentially large receptive field. For example, Fig. 5 compares the network depth-wise effective receptive field of a multi-resolutional network with pooling against that of a reference network without pooling layer. With the same size convolutional filters, the effective receptive field is enlarged in the network with pooling layers. Our multi-resolution analysis (MRA) is indeed derived to supplement the enlarged receptive field from pooling layers with the fine detailed processing using high-pass band convolutional framelets.
In the following, we therefore propose a new class of deep learning network using multi-resolution deep convolutional framelets, which is believed to be quite universal for various inverse problems.

Multi-resolution Analysis.
More specifically, at the first layer, we are interested in learning the local basis Ψ (1) and its dualΨ (1) such that For simplicity we assume that the nonlocal basis Φ is orthonormal. For MRA, we decompose the nonlocal orthonormal basis Φ (1) into the low and high frequency subbands, i.e.
For example, if we use Haar wavelet, the first layer operator Φ low is exactly the same as the average pooling operation in (31), but Φ (1) = Φ now constitutes an orthonormal basis in R n . We also define the approximate signal C (1) low and the detail signal C high : Note that this operation corresponds to the local filtering followed by non-local transform as shown in the red block of Fig. 8. Then, at the first layer, we have the following decomposition: At the second layer, we proceed similarly using the approximate signal C low . More specifically, we are interested in using orthonormal non-local transform: low and Φ (2) high transforms the approximate signal C (1) low ∈ R n/2×d (1) to low and high bands, respectively (see Fig. 8): where p (2) = d (1) denotes the number of Hankel blocks in (6), d (2) is the second layer convolution filter length, and Again, Φ (2) low corresponds to the standard average pooling operation. Note that we need a lifting operation to an extended Hankel matrix with p (2) = d (1) Hankel blocks in (50), because the first layers generates p (2) filtered output which needs to be convolved with d (2) -length filters in the second layer.
Similarly, the approximate signal needs further processing from the following layers. In general, for l = 1, · · · , L, we have where p (l) = p (l−1) d (l−1) denotes the dimension of local basis at l-th layer. This results in L-layer deep convolutional framelets using Haar wavelet.
The multilayer implementation of convolutional framelets now results in an interesting encoderdecoder deep network structure as shown in Fig. 8, where the red and blue blocks represent encoder and decoder blocks, respectively. In addition, Table 1 summarizes the dimension of the l-th layer matrices. More specifically, with the ReLU, the encoder parts are given as follows: which corresponds to multi-channel filtering with local filter Ψ (l) ∈ R p (l) d (l) ×p (l) d (l) followed by the non-local transform Φ low . On the other hand, the decoder part is given by where we use Here, C high is a direct output from the encoder at the same layer, whereasĈ Local basis

Table 1
The nomenclature and dimensions of the matrices at the l-th layer MRA using deep convolutional framelet. Here, d (l) denotes the filter length, and p (l) = p (l−1) d (l−1) with p (0) = d (0) = 1 refers to the number of Hankel block. Fig. 9. A multi-resolution deep convolutional framelet decomposition with a length-2 local filters. Fig. 9 shows the overall structure of multi-resolution analysis with convolutional framelets when length-2 local filters are used. Note that the structure is quite similar to U-net structure [43], except the high pass filter pass. This again confirms a close relationship between the deep convolutional framelets and deep neural networks.

Extension to 2-D Problems.
For the applications of image reconstruction, our multiresolution deep convolutional framelets is extended to 2-D structure, and the proposed architecture is illustrated in Fig. 10. The proposed network is basically obtained using 2-D Haar wavelet. Furthermore, the redundant filter channels are added between each layers, enabling artifact removal of high-band and low-band signals at the same time.
6.1. Methods. In order to evaluate the performance of the proposed deep convolutional framelets for inverse problems, a challenging problems of sparse view x-ray computed tomography (CT) problem is used. More specifically, in X-ray CT, due to the potential risk of radiation exposure, the main research thrust is to reduce the radiation dose. Among various approaches for low-dose CT, sparse-view CT is a recent proposal that reduces the radiation dose by reducing the number of projection views [46]. However, due to the insufficient projection views, standard reconstruction using the filtered back-projection (FBP) algorithm exhibits severe streaking artifacts that are globally distributed. Accordingly, researchers have extensively employed compressed sensing approaches [11] that minimize the the total variation (TV) or other sparsity-inducing penalties under the data fidelity [46]. These approaches are, however, computationally very expensive due to the repeated applications of projection and back-projection during iterative update steps. Therefore, the main goal of this experiment is to apply the proposed network for sparse view CT reconstruction that outperforms the existing approaches in its computational speed as well as reconstruction quality. To address this, our network is trained to learn streaking artifacts as suggested in [17], which is in fact equivalent to learn images with a skipped connection [23]. Once the streaking artifacts are estimated, an artifact-free image is then obtained by subtracting the estimated streaking artifacts as shown in Fig. 10.
As a training data, we used the ten patient data provided by AAPM Low Dose CT Grand Challenge (http://www.aapm.org/GrandChallenge/LowDoseCT/). The data is composed of 3-D CT projection data from 2304 views. Artifact-free original images were generated by FBP using all 2304 projection views. Sparse view reconstruction input images X were generated using FBP from 48, 64, 96, and 192 projection views, respectively. For the proposed residual learning, the label data Y were defined as the difference between the sparse view reconstruction and the full view reconstruction.
Among the ten patient data, eight patient data were used for training, one patient data was used for validation, whereas a test was conducted using the remaining another patient data. This corresponding to 3602 and 504 slices of 512 × 512 images for the training and validation data, respectively, and 488 slices of 512 × 512 images for the test data. The training data was augmented by conducting horizontal and vertical flipping. For the training data set, we used the FBP reconstruction using 48, 96, and 196 projection views simultaneously as input X and the difference between the full view (2304 views) reconstruction and the sparse view reconstructions were used as label Y .
As for the baseline network for comparison, we use the U-net structure in Fig. 7 [17] and a single resolution CNN. The single resolution CNN has the same architecture with U-net in Fig. 7, except that pooling and unpooling were not used. All these network were trained similarly using the same data set. For quantitative evaluation, we use the normalized mean square error (NMSE), which is defined as wherex is the reconstructed X-ray CT images from sparse view and x is a full-view reconstruction image (ground truth). The proposed network was trained by stochastic gradient descent (SGD). The regularization parameter was λ = 10 −4 . The learning rate was set from 10 −3 to 10 −5 which was gradually reduced at each epoch. The number of epoch was 100. A mini-batch data using image patch was used, and the size of image patch was 256 × 256. The network was implemented using MatConvNet toolbox (ver.24) [49] in MATLAB 2015a environment (Mathwork, Natick). We used a GTX 1080 graphic processor and i7-4770 CPU (3.40GHz). The network takes about 4 days for training. 6.2. Reconstruction results. Due to the high-pass branch of the network to guarantee the PR condition, the deep convolutional framelets produced consistently improved images across all view downsampling factor. Fig. 11(a)-(e) shows reconstruction results from 48 projection views. Due to the severe view downsampling, the FBP reconstruction result in Fig. 11(b) provides severely corrupted images with significant streaking artifacts. Accordingly, all the reconstruction results in Fig. 11(c)-(e) were not compatible to the full view reconstruction results in Fig. 11(a). In particular, there are significant remaining streaking artifacts for the conventional CNN architecture (Fig. 11(c)), which were reduced using U-net as shown in Fig. 11(d). However, as indicated by the arrow, some remaining streaking artifacts were visible in Fig. 11(d). On the other hand, the proposed network totally removes the streaking artifact as shown in Fig. 11(e). Quantitative evaluation also showed that the proposed deep convolutional framelets has the minimum NMSE values.
As for reconstruction results from larger number of projection views, Fig. 12(a)-(e) shows reconstruction results from 192 projection views. All the algorithms significantly improved compared to the 48 view reconstruction. However, the reconstruction results by single resolution CNN in Fig.  13(b) and U-net in Fig. 13(c) are somewhat blurry. On the other hand, most of the edge structures were accurately reconstructed by the proposed deep convolutional framelets as shown in Fig. 12(e). Quantitative evaluation also showed that the proposed deep convolutional framelets has the minimum NMSE values. The zoomed area in Fig. 13(a)-(e) also confirmed the findings. The reconstruction result by the deep convolutional framelets provided very realistic image, whereas the other results are somewhat blurry.
These experimental results clearly confirmed that the proposed network is quite universal in the sense it can be used for various artifact patterns. This is due to the network structure retaining the high-pass band to support PR condition, which automatically adapts the resolutions even though various scale of image artifacts are present.

7.
Discussions. While our mathematical theory of deep convolutional framelet was derived for inverse problems, there are many important implications of our finding to general deep learning. For example, we conjecture that the classification network corresponds to a part of our deep convolutional framelets. More specifically, the encoder part of the deep convolutional framelets works for the energy compaction, so the classifier attached to the encoder is basically discriminating the input signals based on the compacted energy distributions from the extended Hankel matrix at the last layer. This is similar with the classical classifier design, where the feature vector is first obtained by a dimensionality reduction algorithm, after which support vector machine (SVM) type classifier is used. Accordingly, the role of the ReLU, residual net, and redundant channels are believed to hold for classifier networks as well. It is also important to note that the singular value spectrum of Hankel matrix, which determines the energy distribution of convolutional framelets coefficients, is translation and rotation invariant as shown in [25]. The invariance property was considered the most important property which gives the theoretical motivation for Mallat's wavelet scattering network [37,3]. Therefore, there may be an important connection between the deep convolutional framelets and wavelet scattering, and our deep convolutional framelets may be effective for classification purpose. However, this is beyond scope of current paper and will be left for future research.
Another interesting observation is that the perfect reconstruction is directly related to finite sample expressivity of a neural network [55]. Recently, there appeared a very intriguing article providing empirical evidences that the traditional statistical learning theoretical approaches fail to explain why large neural networks generalize well in practice [55]. To explain this, the authors showed that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points [55]. We conjecture that the perfect finite sample expressivity is closely related to the perfect reconstruction condition, saying that any finite sample size input can be reproduced perfectly using a neural network. The intriguing link between the PR condition and finite sample expressivity needs further investigation.
Finally, one interesting aspect of our convolutional framelet analysis is the increases of filter channels as shown in (38). While this does not appear to follow the conventional implementation of the convolutional filter channels, there is a very interesting article that provides a strong empirical evidence of our theoretical prediction. In the recent paper on pyramidal residual network [16], the authors gradually increase the feature channels across layers. This design was proven to be an effective means of improving generalization ability. This coincides with our prediction in (38); that is, in order to guarantee the PR condition, the filter channel should increase. This again suggests the theoretical potential of the proposed deep convolutional framelets.
8. Conclusions. In this paper, we propose a general deep learning framework for inverse problems called deep convolutional framelets. Unlike the conventional deep learning approaches that are derived by trial and errors, the proposed network architecture was obtained based on key fundamental theoretical advances we have achieved. First, we show that the deep learning is closely related to the existing theory of annihilating filter-based low-rank Hankel matrix approaches (ALOHA) and convolutional framelets. In particular, our theory was motivated by the observation that when a signal is lifted to a high dimensional Hankel matrix, it usually results in the low-rank structure. Furthermore, the lifted Hankel matrix can be expanded using the so-called convolutional framelets with nonlocal basis convolved with local basis that is usually energy compacting when the underlying Hankel matrix has low rank structure. By extending this idea furthermore, we also derived the perfect reconstruction condition for the deep convolutional framelets. In particular, we showed that the perfect reconstruction is still possible when the framelet coefficients are processed with ReLU. Amazingly, many of the important aspects of the deep learning such as generalization power, residual blocks, redundant channels and CReLU emerge from the PR condition under ReLU.
Our discovery provided theoretical justification of many existing deep learning architecture for inverse problems but also revealed their limitations. Accordingly, we proposed a novel class of deep network using multi-resolution convolutional framelets combined with ReLU, redundant channels as well as residual blocks. Our numerical results showed that the proposed deep convolutional framelets can provide improved reconstruction performance under various conditions. We believe that our discovery opens a new research direction toward the complete understanding of deep neural network.
where the first equality comes from (14) and the second equality is from (12), and the last equality is due to (4). Q.E.D.