{'fraction_non_alphanumeric': 0.029512549788202568, 'fraction_numerical': 0.006436113042928494, 'mean_word_length': 5.094790382244143, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing

Deep neural networks provide unprecedented performance gains in many real world problems in signal and image processing. Despite these gains, future development and practical deployment of deep networks is hindered by their blackbox nature, i.e., lack of interpretability, and by the need for very large training sets. An emerging technique called algorithm unrolling or unfolding offers promise in eliminating these issues by providing a concrete and systematic connection between iterative algorithms that are used widely in signal processing and deep neural networks. Unrolling methods were first proposed to develop fast neural network approximations for sparse coding. More recently, this direction has attracted enormous attention and is rapidly growing both in theoretic investigations and practical applications. The growing popularity of unrolled deep networks is due in part to their potential in developing efficient, high-performance and yet interpretable network architectures from reasonable size training sets. In this article, we review algorithm unrolling for signal and image processing. We extensively cover popular techniques for algorithm unrolling in various domains of signal and image processing including imaging, vision and recognition, and speech processing. By reviewing previous works, we reveal the connections between iterative algorithms and neural networks and present recent theoretical results. Finally, we provide a discussion on current limitations of unrolling and suggest possible future research directions.


I. INTRODUCTION
The past decade has witnessed a deep learning revolution. Availability of large-scale training datasets often facilitated by internet content, accessibility to powerful computational resources thanks to breakthroughs in microelectronics, and advances in neural network research such as development of effective network architectures and efficient training algorithms have resulted in unprecedented successes of deep learning in innumerable applications of computer vision, pattern recognition and speech processing. For instance, deep learning has provided significant accuracy gains in image recognition, one of the core tasks in computer vision. Groundbreaking performance improvements have been demonstrated via AlexNet [1], and lower classification errors than human-level performance [2] was reported on the ImageNet dataset [3].
In the realm of signal processing, learning based approaches provide an interesting algorithmic alternative to traditional V model based analytic methods. In contrast to conventional iterative approaches where the models and priors are typically designed by analyzing the physical processes and handcrafting, deep learning approaches attempt to automatically discover model information and incorporate them by optimizing network parameters that are learned from real world training samples. Modern neural networks typically adopt a hierarchical architecture composed of many layers and comprise a large number of parameters (can be millions), and are thus capable of learning complicated mappings which are difficult to design explicitly. When training data is sufficient, this adaptivity enables deep networks to often overcome model inadequacies, especially when the underlying physical scenario is hard to characterize precisely.
Another advantage of deep networks is that during inference, processing through the network layers can be executed very fast. Many modern computational platforms are highly optimized towards special operations such as convolutions, so that inference via deep networks is usually quite computationally efficient. In addition, the number of layers in a deep network is typically much smaller than the number of iterations required in an iterative algorithm. Therefore, deep learning methods have emerged to offer desirable computational benefits over state-of-the art in many areas of signal processing, imaging and vision. Their popularity has reached new heights with widespread availability of the supporting software infrastructure required for their implementation.
That said, a vast majority of deep learning techniques are purely data-driven and the underlying structures are difficult to interpret. Previous works largely apply general network architectures (some of them will be covered in II-A) towards different problems, and learn certain underlying mappings such as classification and regression functions completely through end-to-end training. It is therefore hard to discover what is learned inside the networks by examining the network parameters, which are usually of high dimensionality, and what are the roles of individual parameters. Interpretability is of course, an important concern both in theory and practice. It is usually the key to conceptual understanding and advancing the frontiers of knowledge. Moreover, in areas such as medical applications and autonomous driving, it is crucial to identify the limitations and potential failure cases of designed systems, where interpretability plays a fundamental role. Thus, lack of interpretability can be a serious limitation of conventional deep learning methods in contrast with model-based techniques with iterative algorithmic solutions which are used widely in signal processing.
An issue that frequently arises together with interpretability arXiv:1912.10557v1 [eess.IV] 22 Dec 2019 is generalizability. It is well known that the practical success of deep learning is sometimes overly dependent on the quantity and quality of training data available. In scenarios where abundant high-quality training samples are unavailable such as medical imaging [4], [5] and 3D reconstruction [6], the performance of deep networks may degrade significantly, and sometimes may even underperform traditional approaches. This phenomenon, formally called overfitting in machine learning terminology, is largely due to the employment of generic neural networks that are associated with a huge number of parameters. Without exploiting domain knowledge explicitly beyond time and/or space invariance, such networks are highly under regularized and may overfit severely even under heavy data augmentations.
To some extent, this problem has recently been addressed via the design of domain enriched or prior information guided deep networks [7], [8], [9]. In such cases, the network architecture is designed to contain special layers that are domain specific, such as the transform layer in [8]. In other cases, prior structure of the expected output is exploited [8], [9] to develop training robustness. An excellent tutorial article covering these issues is [10]. Despite these achievements, transferring domain knowledge to network parameters can be a nontrivial task and effective priors may be hard to design. More importantly, the underlying network architectures in these approaches largely remain consistent with conventional neural networks. Therefore, the pursuit of interpretable, generalizable and high performance deep architectures for signal processing problems remains a highly important open challenge.
In the seminal work of Gregor and LeCun [11], a promising technique called algorithm unrolling (or unfolding) was developed that has helped connect iterative algorithms such as those for sparse coding to neural network architectures. Following this, the past few years have seen a surge of efforts that unroll iterative algorithms for many significant problems in signal and image processing: examples include (but are not limited to) compressive sensing [12], deconvolution [13] and variational techniques for image processing [14]. Figure 1 provides a high-level illustration of this framework. Specifically, each iteration of the algorithm step is represented as one layer of the network. Concatenating these layers forms a deep neural network. Passing through the network is equivalent to executing the iterative algorithm a finite number of times. In addition, the algorithm parameters (such as the model parameters and regularization coefficients) transfer to the network parameters. The network may be trained using back-propagation resulting in model parameters that are learned from real world training sets. In this way, the trained network can be naturally interpreted as a parameter optimized algorithm, effectively overcoming the lack of interpretability in most conventional neural networks.
Traditional iterative algorithms generally entail significantly fewer number of parameters compared with popular neural networks. Therefore, the unrolled networks are highly parameter efficient and require less training data. In addition, the unrolled networks naturally inherit prior structures and domain knowledge rather than learn them from intensive training data. Consequently, the unrolled networks tend to generalize better than generic networks, and can be computationally faster as long as each algorithmic iteration (or the corresponding layer) is not overly expensive.
In this article, we review the foundations of algorithm unrolling. Our goal is to provide readers with guidance on how to utilize unrolling to build efficient and interpretable neural networks in solving signal and image processing problems. After providing a tutorial on how to unroll iterative algorithms into deep networks, we extensively review selected applications of algorithm unrolling in a wide variety of signal and image processing domains. We also review general theoretical studies that shed light on convergence properties of these networks, although further analysis is an important problem for future research. In addition, we clarify the connections between general classes of traditional iterative algorithms and deep neural networks established through algorithm unrolling. We contrast algorithm unrolling with alternative approaches and discuss their strengths and limitations. Finally, we discuss current limitations and open challenges, and suggest future research directions.

II. GENERATING INTERPRETABLE NETWORKS THROUGH ALGORITHM UNROLLING
We begin by describing algorithm unrolling. To motivate the unrolling approach, we commence with a brief review on conventional neural network architectures in Section II-A. We next discuss the first unrolling technique for sparse coding in Section II-B. We elaborate on general forms of unrolling in Section II-C.

A. Conventional Neural Networks
In early neural network research the Multi-Layer Perceptrons (MLP) was a popular choice. This architecture can be motivated either biologically by mimicking the human recognition system, or algorithmically by generalizing the perceptron algorithm to multiple layers. A diagram illustration of MLP is provided in Fig. 2a. The network is constructed through recursive linear and nonlinear operations, which are called layers. The units those operations act upon are called neurons, which is an analogy of the neurons in the human nerve systems. The first and last layer are called input layer and output layer, respectively.
A salient property of this network is that each neuron is fully connected to every neuron in the previous layer, except for the input layer. The layers are thus commonly called Fully-Connected layers. Analytically, in the l-th layer the relationship between the neurons x l j and x l+1 i is expressed as where W l+1 and b l+1 are the weights and biases, respectively, and σ is a nonlinear activation function. We omit drawing activation functions and biases in Fig. 2a  . Each iteration h depends on algorithm parameters θ, which are transferred into network parameters θ 1 , θ 2 , . . . . Instead of determining these parameters through cross-validation or analytical derivations, we learn θ 1 , θ 2 , . . . from training datasets through end-to-end training. In this way, the resulting network could achieve better performance than the original iterative algorithm. In addition, the network layers naturally inherit interpretability from the iteration procedure. The learnable parameters are colored in blue.
the hyperbolic tangent function. In recent years, they have been superseded by Rectified Linear Units (ReLU) [15] defined by The W's and b's are generally trainable parameters that are learned from datasets through training, during which backpropagation [16] is often employed for gradient computation.
Nowadays, MLPs are rarely seen in practical imaging and vision applications. The fully-connected nature of MLPs contributes to a rapid increase in their parameters, making them difficult to train. To address this limitation, LeCun [17] proposed to restrict neuron connections to local neighbors only and let weights be shared across different spatial locations. The linear operations then become convolutions (or correlations in a strict sense) and thus the networks employing such localizing structures are generally called Convolutional Neural Networks (CNN). A visual illustration of a CNN can be seen in Fig. 2b. With significantly reduced parameter dimensionality, training deeper networks becomes feasible. While CNNs were first applied to digit recognition, their translation invariance is a desirable property in many computer vision tasks. CNNs thus have become an extremely popular and indispensable architecture in imaging and vision, and outperform traditional approaches by a large margin in many domains. Today they continue to exhibit the best performance in many applications.
In applications such as speech recognition and video processing, where data comes sequentially, Recurrent Neural Networks (RNN) [18] are a popular choice. RNNs explicitly model the data dependence in different time steps in the sequence and scale well to sequences with varying lengths. A visual depiction of RNNs is provided in Fig. 2c. Given the previous hidden state s l−1 and input variable x l , the next hidden state s l is computed as while the output variable o l is generated by Here U, V, W, b 1 , b 2 are trainable network parameters and σ 1 , σ 2 are activation functions. We again omit the activation functions and biases in Fig. 2c. In contrast to MLPs and CNNs where the layer operations are applied recursively in a hierarchical representation fashion, RNNs apply the recursive operations as the time step evolves. A distinctive property of RNNs is that the parameters U, V, W are shared across all the time steps, rather than varying from layer to layer. Training RNNs can thus be difficult as the gradients of the parameters may either explode or vanish.

B. Unrolling Sparse Coding Algorithms into Deep Networks
The earliest work in algorithm unrolling dates back to Gregor et al.'s paper on improving the computational efficiency of sparse coding algorithms through end-to-end training [11]. In particular, they discussed how to improve the efficiency of the Iterative Shrinkage and Thresholding Algorithm (ISTA), one of the most popular approaches in sparse coding. The crux of this work is summarized in Fig. 3 and detailed in the box on Learned ISTA. Each iteration of ISTA comprises one linear operation followed by a non-linear soft-thresholding operation, which mimics the ReLU activation function. A diagram representation of one iteration step reveals its resemblance to a single network layer. Thus, one can form a deep network by mapping each iteration to a network layer and stacking

Input Layer
Hidden Layers Output layer

Input Layer Hidden Layers
Output layer Hidden States

Inputs
Outputs the layers together which is equivalent to executing an ISTA iteration multiple times. Because the same parameters are shared across different layers, the resulting network resembles RNN in terms of architecture.
After unrolling ISTA into a network, the network is trained using training samples through back-propagation. The learned network is dubbed Learned ISTA (LISTA). It turns out that significant computational benefits can be obtained by learning from real data. For instance, Gregor et al. [11] experimentally verified that the learned network reaches a specific performance level around 20 times faster than accelerated ISTA. Consequently, the sparse coding problem can be solved efficiently by passing through a compact LISTA network. From a theoretical perspective, recent studies [19], [20] have characterized the linear convergence rate of LISTA and further verified its computational advantages in a rigorous and quantitative manner. A more detailed exposition and discussion on related theoretical studies will be provided in Section IV-D.
In addition to ISTA, Gregor et al. discussed unrolling and optimizing another sparse coding method, the Coordinate Descent (CoD) algorithm [21]. The technique and implications of unrolled CoD are largely similar to LISTA.

C. Algorithm Unrolling in General
Although Gregor et al.'s work [11] focused on improving computational efficiency of sparse coding, the same techniques can be applied to general iterative algorithms. An illustration is given in Fig. 4. In general, the algorithm repetitively performs certain analytic operations, which we represent abstractly as the h function. Similar to LISTA, we unroll the algorithm into a deep network by mapping each iteration into a single network layer and stacking a finite number of layers together. Each iteration step of the algorithm contains parameters, such as the model parameters or the regularization coefficients, which we denote by vectors θ l , l = 0, . . . , L − 1. Through unrolling, the parameters θ l correspond to those of the deep network, and can be optimized towards real-world scenarios by training the network end-to-end using real datasets.
While the motivation of LISTA was computational savings, proper use of algorithm unrolling can also lead to dramatically improved performance in practical applications. For instance, we can employ back-propagation to obtain coefficients of filters [13] and dictionaries [22] that are hard to design either analytically or even by handcrafting. In addition, custom modifications may be employed in the unrolled network [12]. As a particular example, in LISTA (see the box Learned ISTA), the matrices W t , W e may be learned in each iteration, so that they are no longer held fixed throughout the network. By allowing a slight departure from the original iterative algorithms [11], [23] and extending the representation capacity, the performance of the unrolled networks may be boosted significantly.
Compared with conventional generic neural networks, unrolled networks generally contain significantly fewer parameters, as they encode domain knowledge through unrolling. In addition, their structures are more specifically tailored towards target applications. These benefits not only ensure higher efficiency, but also provide better generalizability especially under limited training schemes [12]. More concrete examples will be presented and discussed in Section III.

III. UNROLLING IN SIGNAL AND IMAGE PROCESSING PROBLEMS
Algorithm unrolling has been applied to diverse application areas in the past few years. Table I summarizes representative methods and their topics of focus in different domains. Evidently, research in algorithm unrolling is growing and influencing a variety of high impact real-world problems and research areas. As discussed in Section II, an essential element of each unrolling approach is the underlying iterative algorithm it starts from, which we also specify in Table I.

Learned ISTA
The pursuit of parsimonious representation of signals has been a problem of enduring interest in signal processing. One of the most common quantitative manifestations of this is the well-known sparse coding problem [24]. Given an input vector y ∈ R n and an over-complete dictionary W ∈ R n×m with m > n, sparse coding refers to the pursuit of a sparse representation of y using W. In other words, we seek a sparse code x ∈ R m such that y ≈ Wx while encouraging as many coefficients in x to be zero (or small in magnitude) as possible. A well-known approach to determine x is to solve the following convex optimization problem: where λ > 0 is a regularization parameter that controls the sparseness of the solution. A popular method for solving (2) is the family of Iterative Shrinkage and Thresholding Algorithms (ISTA) [25]. In its simplest form, ISTA performs the following iterations: where I ∈ R m×m is the identity matrix, S λ (·) is the softthresholding operator defined elementwise as and µ > 0 is a parameter that controls the step size. Basically, ISTA is equivalent to a gradient step of y −Wx 2 2 , followed by a projection onto the 1 ball. As depicted in Fig. 3, the iteration (3) can be recast into a single network layer. This layer comprises a series of analytic operations (matrix-vector multiplication, summation, soft-thresholding), which is of the same nature as a neural network. Executing ISTA L times can be interpreted as cascading L such layers together, which essentially forms an L-layer deep network. In the unrolled network an implicit substitution of parameters has been made: W t = I − 1 µ W T W and W e = 1 µ W T . While these substitutions generalize the original parametrization and expand the representation power of the unrolled network, recent theoretical studies [20] suggest that they may be inconsequential in an asymptotic sense as the optimal network parameters admit a weight coupling scheme asymptotically. The unrolled network is trained using real datasets to optimize the parameters W t , W e and λ. The learned version, LISTA, may achieve higher efficiency compared to ISTA. It is also useful when W is not known exactly. Training is performed using a sequence of vectors y 1 , y 2 , . . . , y N ∈ R n and their corresponding groundtruth sparse codes x * 1 , x * 2 , . . . , x * N . By feeding each y n , n = 1, . . . , N into the network, we retrieve its output x n (y n ; W t , W e , λ) as the predicted sparse code for y n . Comparing it with the ground truth sparse code x * n , the network training loss function is formed as: and the network is trained through loss minimization, using popular gradient-based learning techniques such as stochastic gradient descent [16], to learn W t , W t , λ. It has been shown empirically that, the number of layers L in (trained) LISTA can be an order of magnitude smaller than the number of iterations required for ISTA [11] to achieve convergence corresponding to a new observed input.

A. Applications in Computational Imaging
Computational imaging is a broad area covering a wide range of interesting topics, such as computational photography, hyperspectral imaging, and compressive imaging, to name a few. The key to success in many computational imaging areas frequently hinges on solving an inverse problem. Modelbased inversion has long been a popular approach. Examples of model-based methods include parsimonious representations such as sparse coding and low-rank matrix pursuit, variational methods and conditional random fields. The employment of model-based techniques gives rise to many iterative approaches, as closed-form solutions are rarely available. The fertile ground of these iterative algorithms in turn provides a solid foundation and offers many opportunities for algorithm unrolling.
Single image super-resolution is an important topic in computational imaging that focuses on improving the spatial resolution of a single degraded image. In addition to offering images of improved visual quality, super-resolution also aids diagnosis in medical applications and promises to improve the performance of recognition systems. Compared with naive bicubic interpolation, there exists large room for performance improvement by exploiting natural image structures, such as learning dictionaries to encode local image structures into sparse codes [36]. Significant research effort has been devoted towards structure-aware approaches. Wang et al. [22] applied LISTA (which we discussed in Section II-B) to patches extracted from the input image, and recombine the predicted high-resolution patches to form a predicted high-resolution image. A diagram depiction of the entire network architecture, dubbed Sparsity Coding based Network (SCN), is provided in Fig. 5. Trainable parameters of SCN include those of LISTA (W t , W e , λ) and the dictionary D. By integrating the patch extraction and recombination layers into the network, the whole network resembles a CNN as it also performs patch-by-patch processing. The network is trained by pairing   [22] architecture: the patches extracted from input low-resolution image y are fed into a LISTA sub-network to estimate the associated sparse codes α, and then high-resolution patches are reconstructed through a linear layer. The predicted high-resolution image x is formed by putting these patches into their corresponding spatial locations. The whole network is trained by forming low-resolution and high-resolution image pairs, using standard stochastic gradient descent algorithm. The high resolution dictionary D (colored in blue) and the LISTA parameters are trainable from real datasets.   low-resolution and high-resolution images and minimizing the Mean-Square-Error (MSE) loss 1 . In addition to higher visual quality and PSNR gain of 0.3dB to 1.6dB over state-of-the art, the network is faster to train and has reduced number of parameters. Fig. 6 provides sample visual results from SCN and several state-of-the art techniques. Another important application focusing on improving the quality of degraded images is blind image deblurring. Given a sharp image blurred by an unknown function, which is usually called the blur kernel or Point Spread Function, the goal is to jointly estimate both the blur kernel and the underlying sharp image. There is a wide range of approaches in the literature to blind image deblurring: the blur kernel can be of different forms, such as Gaussian, defocusing and motion. In addition, the blur kernel can be either spatially uniform or non-uniform.

Iterative Shrinkage and Thresholding Algorithm A Single Network Layer
Blind image deblurring is a challenging topic because the blur kernel is generally of low-pass nature, rendering the problem highly ill-posed. Most existing approaches rely on extracting stable features (such as salient edges in natural images) to reliably estimate the blur kernels. The sharp image can be retrieved subsequently based on the estimated kernels. Schuler et al. [28] reviews many existing algorithms and note that they essentially iterate over three modules: 1) feature extraction, 2) kernel estimation and 3) image recovery.
Therefore, a network can be built by unrolling and concatenating several layers of these modules, as depicted in Fig. 7. More specifically, the feature extraction module is modeled as a few layers of convolutions and rectifier operations, which basically mimics a small CNN, while both the kernel and image estimation modules are modeled as least-square operations. To train the network, the blur kernels are simulated by sampling from Gaussian processes, and the blurred images are synthetically created by blurring each sharp image through a 2D discrete convolution.
Recently, Li et al. [13], [33] develop an unrolling approach for blind image deblurring by enforcing sparsity constraints over filtered domains and then unrolling the half-quadratic splitting algorithm for solving the resulting optimization problem. The network is called Deep Unrolling for Blind Deblurring (DUBLID) and detailed in the DUBLID box. As the DUBLID box reveals, custom modifications are made in the unrolled network to integrate domain knowledge that enchanes the deblurring pursuit. The authors also derive custom backpropagation rules analytically to facilitate network training. Experimentally, the network offers significant performance gains and requires much fewer parameters and less inference time compared with both traditional iterative algorithms and modern neural network approaches. An example of experimental comparisons is provided in Fig. 9.

B. Applications in Medical Imaging
Medical imaging is a broad area that generally focuses on applying image processing and pattern recognition techniques to aid clinical analysis and disease diagnosis. Interesting topics in medical imaging include Medical Resonance Imaging (MRI), Computed Tomography (CT) imaging, Ultrasound (US) imaging, to name a few. Just like computational imaging, medical imaging is an area enriched with many interesting inverse problems, and model-based approaches such as sparse coding play a critical role in solving these problems. In practice, data collection can be quite expensive and painstaking for the patients, and therefore it is difficult to collect abundant samples to train conventional deep networks. Interpretability is also an important concern. Therefore, algorithm unrolling has great potential in this context.

Deep Unrolling for Blind Deblurring (DUBLID)
The spatially invariant blurring process can be represented as a discrete convolution: where y is the blurred image, x is the latent sharp image, k is the unknown blur kernel, and n is Gaussian random noise. A popular class of image deblurring algorithms perform Total-Variation (TV) minimization, which solves the following optimization problem: where D x y, D y y are the partial derivatives of y in horizontal and vertical directions respectively, and λ 1 , λ 2 , ε are positive regularization coefficients. Upon convergence, the variables g 1 and g 2 are estimates of the sharp image gradients in the x and y directions, respectively. In [13], [33] (7) was generalized by realizing that D x and D y are computed using linear filters which can be generalized into a set of C filters An efficient optimization algorithm to solving (8) is the Half-Quadratic Splitting Algorithm, which alternately minimizes the surrogate problem over the variables and k sequentially. Here ζ i , i = 1, . . . , C are regularization coefficients. A noteworthy fact is that each individual minimization admits an analytical expression, which facilitates casting (9) into network layers. Specifically, in the l-th iteration (l ≥ 0) the following updates are performed: where [·] + is the ReLU operator, x denotes the Discrete Fourier Transform (DFT) of x, F −1 indicates the inverse DFT operator, refers to elementwise multiplication, S is the soft-thresholding operator defined elementwise in (4), and the operator N 1 (·) normalizes its operand into unit sum. Here stacked together. Note that layer-specific parameters ζ l , β l , f l are used. As with most existing unrolling methods, only L iterations are performed. The sharp image is retrieved from g L and k L by solving the following linear least-squares problem: are the filter coefficients in the L-th layer, and η = {η i } C i=1 are positive regularization coefficients. By unrolling (10) and (11) into a deep network, we get L layers of g, z and k updates, followed by one layer of image retrieval. The filter coefficients f l i 's and regularization parameters {λ l i , ζ l i , η i }'s are learned by back-propagation. Note that f L i 's are shared in both (10) and (11) and are updated jointly. The final network architecture is depicted in Fig. 8. Similar to [28], the network is trained using synthetic samples, i.e., by convolving the sharp images to obtain blurred versions. The training loss function is the translationinvariant MSE loss to compensate for the possible spatial shifts of the deblurred images and the blur kernel.
In MRI, a fundamental challenge is to recover a signal from a small number of its measurements, corresponding to reduced scanning time. To this end, Yang et al. [12] unroll the widely-known Alternating Direction Method of Multipliers (ADMM) algorithm, a popular optimization algorithm for solving CS and related sparsity-constrained estimation problems, into a deep network called ADMM-CSNet. The sparsityinducing transformations and regularization coefficients are learned from real data to advance its limited adaptability and enhance the reconstruction performance. Compared with conventional iterative methods, ADMM-CSNet achieves the same reconstruction accuracy using 10% less sampled data and speeds up recovery by around 40 times. It exceeds stateof-the art deep networks by around 3dB PSNR under 20% sampling rate. Refer to the box on ADMM-CSNet for further details.

ADMM-CSNet
Given random linear measurements y ∈ C m formed by y ≈ Φx where Φ ∈ C m×n is a measurement matrix with m < n, Compressive Sensing (CS) aims at reconstructing the original signal x ∈ R n by exploiting its underlying sparse structure in a transform domain [24]. A generalized CS model can be formulated as the following optimization problem [12]: where λ i 's are positive regularization coefficients, g(·) is a sparsity-inducing function, and is a sequence of C Toeplitz operators, which effectively perform linear filtering operations. Concretely, D i can be taken as a wavelet transform and g can be chosen as the 1 norm. However, for better performance the method in [12] learns both of them from an unrolled network. An efficient minimization algorithm for solving (12) is the Alternating Direction Method of Multipliers (ADMM) [40]. Problem (12) is first recast into a constrained minimization through variable splitting: The corresponding augmented Lagrangian is then formed as follow: where are penalty coefficients. ADMM then alternately minimizes (14) followed by a dual variable update, leading to the following iterations: where η i 's are constant parameters, and P g {·; λ} is the proximal mapping for g with parameter λ. The unrolled network can thus be constructed by concatenating these operations and learning the parameters λ i , ρ i , η i , D i in each layer. Fig. 10 depicts the resulting unrolled network architecture. In [12] the authors discuss several implementation issues, including efficient matrix inversion and the back-propagation rules. The network is trained by minimizing a normalized version of the Root-Mean-Square-Error.
Another important imaging modality is ultrasound, which has the advantage of being a radiation-free approach. When used for blood flow depiction, one of the challenges is the fact that the tissue reflections tend to be much stronger than those of the blood, leading to strong clutter resulting from the tissue. Thus, an important task is to separate the tissue from the blood. Various filtering methods have been used in this context such as high pass filtering, and filtering based on the singular value decomposition. Solomon et al. [30] suggest using a robust Principal Component Analysis (PCA) approach by modeling the received ultrasound movie as a low-rank and sparse matrix where the tissue is low rank and the blood vessels are sparse. They then unroll an ISTA approach to robust PCA into a deep network, called Convolutional rObust pRincipal cOmpoNent Analysis (CORONA). As the name suggests, they replace matrix multiplications with convolutional layers, effectively converting the network into a CNN-like architecture. Compared with state of-the art approaches, CORONA demonstrates vastly improved reconstruction quality and has much fewer parameters than the well-known ResNet [41]. Refer to  (10) and (11) into the network. Trainable parameters are colored in blue.  . . . U 1 (·, ·, ·; ρ, D) U 2 (·, ·; λ, ρ, D U 3 (·, ·, ·; η, D) Fig. 10. Diagram representation of ADMM-CSNet [12]: each stage comprises a series of inter-related operations, whose analytic forms are given in (15). Note that the trainable parameters are colored in blue. the box on CORONA for details.

C. Applications in Vision and Recognition
Computer vision is a broad and fast growing area that has achieved tremendous success in many interesting topics in recent years. A major driving force for its rapid progress is deep learning. For instance, thanks to the availability of large scale training samples, in image recognition tasks researchers have surpassed human-level performance over the ImageNet dataset by employing deep CNN [2]. Nevertheless, most existing approaches to date are highly empirical, and lack of interpretability has become an increasingly serious issue. To overcome this drawback, recently researchers are paying more attention to algorithm unrolling [27], [29].
One example is in semantic image segmentation, which assigns class labels to each pixel in an image. Compared with traditional low-level image segmentation, it provides additional information about object categories, and thus creates semantically meaningful segmented objects. By performing pixellevel labeling, semantic segmentation can also be regarded as an extension to image recognition. Applications of semantic segmentation include autonomous driving, robot vision, and medical imaging.
Traditionally Conditional Random Field (CRF) was a popular approach. Recently deep networks have become the primary tool. Early works of deep learning approaches are capable of recognizing objects at a high level; however, they are relatively less accurate in delineating the objects than CRFs.
Zheng et al. [27] unroll the Mean-Field (MF) iterations of CRF into a RNN, and then concatenate the semantic segmentation network with this RNN to form a deep network. The concatenated network resembles conventional semantic segmentation followed by CRF-based post-processing, while end-to-end training can be performed over the whole network. Liu et al. [29] follow the same direction to construct their segmentation network, called Deep Parsing Network. In their approach, they adopt a generalized pairwise energy and perform MF iteration only once for the purposes of efficiency. Refer to the box on Unrolling CRF into RNN for further details.

D. Applications in Speech Processing
Speech processing is one of the fundamental applications of digital signal processing. Topics in speech processing include recognition, coding, synthesis, and more. Among all problems of interest, source separation stands out as a challenging yet intriguing one. Applications of source separation include speech enhancement and recognition.
For single channel speech separation, Non-negative Matrix Factorization (NMF) was a widely-applied technique. Recently, Hershey et al. [26] unrolled NMF into a deep network, dubbed deep NMF, as a concrete realization of their abstract unrolling framework. Detailed descriptions are in the box Deep NMF. The deep NMF was evaluated on the task of speech enhancement in reverberated noisy mixtures, using a dataset collected from the Wall Street Journal. Deep NMF was shown to outperform both a conventional deep neural network [26] and the iterative sparse NMF method [42].
Wang et al. [32] propose an end-to-end training approach for speech separation, by casting the commonly employed forward and inverse Short-time Fourier Transform (STFT) operations into network layers, and concatenate them with an iterative phase reconstruction algorithm, dubbed Multiple Input Spectrogram Inversion [43]. In doing so, the loss function acts on reconstructed signals rather than their STFT magnitudes, and phase inconsistency can be reduced through training. The trained network exhibits more than 1dB higher Signal-to-Distortion Ratio over state-of-the art techniques on public datasets.

Convolutional Robust Principal Component Analysis
In US imaging, a series of pulses are transmitted into the imaged medium, and their echoes are received in each transducer element. After beamforming and demodulation, a series of movie frames are acquired. Stacking them together as column vectors leads to a data matrix D ∈ C m×n , which can be modeled as follows: where L comprises the tissue signals, S comprises the echoes returned from the blood signals, H 1 , H 2 are measurement matrices, and N is the noise matrix. Due to its high spatial-temporal coherence, L is typically a low-rank matrix, while S is generally a sparse matrix since blood vessels usually sparsely populate the imaged medium. Based on these observations, the echoes S can be estimated through a transformed low-rank and sparse decomposition by solving the following optimization problem: where · * is the nuclear norm of a matrix which promotes low-rank solutions, and · 1,2 is the mixed 1,2 norm which enforces row sparsity. Problem (16) can be solved using a generalized version of ISTA in the matrix domain, by utilizing the proximal mapping corresponding to the nuclear norm and mixed 1,2 norm. In the l-th iteration it executes the following steps: where T λ {X} is the singular value thresholding operator that performs soft-thresholding over the singular values of X with threshold λ, S 1,2 λ performs row-wise softthresholding with parameter λ, and µ is the step size parameter for ISTA. Technically, T λ and S 1,2 λ correspond to the proximal mapping for the nuclear norm and mixed 1,2 norm, respectively. Just like the migration from MLP to CNN, the matrix multiplications can be replaced by convolutions, which gives rise to the following iteration steps instead: where * is the convolution operator. Here P l i , i = 1, . . . , 6 are a series of convolution filters that are learned from the data in the l-th layer, and λ l 1 , λ l 2 are thresholding parameters for the l-th layer. By casting (17) and (18) into network layers, a deep network resembling CNN is formed. The parameters P l i , i = 1, 2, . . . , 6 and {λ l 1 , λ l 2 } are learned from training data. To train the network, one can first obtain ground-truth L and S from D by executing ISTA-like algorithms up to convergence. Simulated samples can also be added to address lack of training samples. MSE losses are imposed on L and S, respectively.

E. Enhancing Efficiency Through Unrolling
In addition to intepretability and performance improvements, unrolling can provide significant advantages for practical deployment, including higher computational efficiency and lower number of parameters, which in turn lead to reduced memory footprints and storage requirements. Table II summarizes selected results from recent unrolling papers to illustrate such benefits. For comparison, results for one iterative algorithm and one deep network are included, both selected from representative top-performing methods. Note further, that for any two methods compared in Table II, the run times are reported on consistent implementation platforms. More details can be found in the respective papers.
Compared to their iterative counterparts, unrolling often dramatically boosts the computational speed. For instance, it was reported in [16] that LISTA may be 20 times faster than ISTA after training; DUBLID [33] can be 1000 times faster than TV-based deblurring; ADMM-CSNet [12] can be about four times faster than the BM3D-AMP algorithm [44]; and CORONA [30] is over 50 times faster than the Fast-ISTA algorithm.
Typically, by embedding domain-specific structures into the network, the unrolled networks need much fewer parameters than conventional networks that are less specific towards particular applications. For instance, the number of parameters for DUBLID is more than 100 times lower than SRN [ while CORONA [30] has an order of magnitude lower number of parameters than ResNet [41]. Under circumstances where each iteration (layer) can be executed highly efficiently, the unrolled networks may even be more efficient than conventional networks. For instance, ADMM-CSNet [12] has shown to be about twice as fast as ReconNet [45], while DUBLID [29] is almost two times faster than DeblurGAN [46].

Unrolling CRF into RNN
CRF is a fundamental model for labeling undirected graphical models. A special case of CRF, where only pairwise interactions of graph nodes are considered, is the Markov random field. Given a graph (V, E) and a predefined collection of labels L, it assigns label l p ∈ L to each node p by minimizing the following energy function: where φ p (·) and ψ p,q (·) are commonly called unary energy and pairwise energy, respectively. Typically, φ p models the preference of assigning p with each label given the observed data, while ψ p,q models the smoothness between p and q. In semantic segmentation, V comprises the image pixels, E is the set of pixel pairs, while L consists of object categories, respectively. In [27], the unary energy φ p is chosen as the output of a semantic segmentation network, such as the wellknown Fully Convolutional Network (FCN) [47], while the pairwise energy ψ p,q (f p , f q ) admits the following special form: where {G m } M m=1 is a collection of Gaussian kernels and {w m } M m=1 are the corresponding weights. Here f p and f q are the feature vectors for pixel p and q, respectively, and µ(·, ·) models label compatibility between pixel pairs. An efficient inference algorithm for energy minimization over fully-connected CRFs is the MF iteration [48], which executes the following steps iteratively: where Q p (l p ) can be interpreted as the margin probability of assigning p with label l p . A noteworthy fact is that each update step resembles common neural network layers. For instance, message passing can be implemented by filtering through Gaussian kernels, which imitates passing through a convolutional layer. The compatibility transform can be implemented through 1 × 1 convolution, while the normalization can be considered as the popular softmax layer. These layers can thus be unrolled to form a RNN, dubbed CRF-RNN. By concatenating FCN with it, a network which can be trained end-to-end is formed. A diagram illustration is in presented Fig. 12.

end for
Activation function Fig. 13. A MLP can be interpreted as executing an underlying iterative algorithm with finite iterations and layer-specific parameters.

Deep (Unrolled) NMF
Single channel source separation refers to the task of decoupling several source signals from their mixture. Suppose we collect a sequence of T mixture frames, where m t ∈ R F + , t = 1, 2, . . . , T is the t-th frame. Given a set of non-negative basis vectors {w l ∈ R F + } L l=1 , we can represent m t (approximately) by where h lt 's are the coefficients chosen to be non-negative. By stacking m t 's column by column, we form a nonnegative matrix M ∈ R F ×T + , so that (21) can be expressed in matrix form: where W has w l as its l-th column, ≥ 0 denotes elementwise non-negativity, and H = (h l,t ). To remove multiplicative ambiguity, it is commonly assumed that each column of W has unit 2 norm, i.e., w l 's are unit vectors. The model (22) is commonly called Non-negative Matrix Factorization (NMF) [49] and has found wide applications in signal and image processing. In practice, the non-negativity constraints prevent mutual cancelling of basis vectors and thus encourage semantically meaningful decompositions, which turns out highly beneficial. Assuming the phases among different sources are approximately the same, the power or magnitude spectrogram of the mixture can be decomposed as a summation of those from each sources. Therefore, after performing NMF, the sources can be separated by selecting basis vectors corresponding to each individual source and recombining the source-specific basis vectors to recover the magnitude spectrograms. In practical implementation, typically a filtering process similar to the classical Wiener filtering is performed for magnitude spectrogram recovery.
To determine W and H from M, one may consider solving the following optimization problem [50]: where D β is the β-divergence, which can be considered as a generalization of the well-known Kullback-Leibler divergence, and µ is a regularization parameter that controls the sparsity of the coefficient matrix H. By employing a majorization-minimization scheme, problem (23) can be solved by the following multiplicative updates: Normalize W l so that the columns of W l have unit norm and scale H l accordingly, (26) for l = 1, 2, . . . . In [26], a slightly different update scheme for W was employed to encourage the discriminative power. We omit discussing it for brevity. A deep network can be formed by unfolding these iterative updates. In [26], W l 's are untied from the update rule (25) and considered trainable parameters. In other words, only (24) and (26) are executed in each layer. Similar to (23), the β-divergence, with a different β value, was employed in the training loss function. A splitting scheme was also designed to preserve the non-negativity of W l 's during training.
IV. CONNECTIONS TO SIGNAL PROCESSING METHODS In addition to creating efficient and interpretable network architectures which achieve superior performance in practical applications, algorithm unrolling can provide valuable insights from a conceptual standpoint. As detailed in the previous Section, solutions to real-world signal processing problems often exploit domain specific prior knowledge. Inheriting this domain knowledge is of both conceptual and practical importance in deep learning research. To this end, algorithm unrolling can potentially serve as a powerful tool to help establish conceptual connections between prior-information guided analytical methods and modern neural networks.
In particular, algorithm unrolling may be utilized in the reverse direction: instead of unrolling a particular iterative algorithm into a network, we can interpret a conventional neural network as a certain iterative algorithm to be identified. Fig. 13 provides a visual illustration of applying this technique to MLP. The same technique is applicable to other networks, such as CNN or RNN.
By interpreting popular network architectures as conventional iterative algorithms, better understanding of the network behavior and mechanism can be obtained. Furthermore, rigorous theoretical analysis of designed networks may be facilitated once an equivalence is established with a wellunderstood class of iterative algorithms. Finally, architectural enhancement and performance improvements of the neural networks may result from incorporating domain knowledge associated with iterative algorithms.

Neural Network Training Using Extended Kalman Filter
The Kalman filter is a fundamental technique in signal processing with a wide range of applications. It obtains the Minimum Mean-Square-Error (MMSE) estimation of system state by recursively drawing observed samples and updating the estimate. The Extended Kalman Filter (EKF) extends to the nonlinear case through iterative linearization. Previous studies [52] have revealed that EKF can be employed to facilitate neural network training, by realizing that neural network training is essentially a parameter estimation problem. More specifically, the training samples may be treated as observations, and if the MSE loss is chosen then network training essentially performs MMSE estimation conditional on observations. Let {(x 1 , y 1 ), (x 2 , y 2 ), . . . , (x N , y N )} be a collection of training pairs. We view the training samples as sequentially observed data following a time order. At time step k, when feeding x k into the neural network with parameters w, it performs a nonlinear mapping h k (·; w k ) and outputs an estimate y k of y k . This process can be formally described as the following nonlinear state-transition model: where ω k and ν k are zero-mean white Gaussian noises with covariance E(ω k ω T l ) = δ k,l Q k and E(ν k ν T l ) = δ k,l R k , respectively. Here E is the expectation operator and δ is the Kronecker delta function. In (27), the noise ω is added artificially to avoid numerical divergence and poor local minima [52]. For a visual depiction, refer to Fig. 14. The state-transition model (27) and (28) is a special case of the state-space model of EKF and thus we can apply the EKF technique to estimate the network parameters w k sequentially. To begin with, at k = 0, w 0 and P 0 are initialized to certain values. At time step k (k ≥ 0), the nonlinear function h k is linearized as: where . For a neural network, H k is essentially the derivative of its output y k over its parameters w k and therefore can be computed via backpropagation. The following recursion is then executed: where K k is commonly called the Kalman gain. For details on deriving the update rules (30), see [53,Chapter 1]. In summary, neural networks can be trained with EKF by the following steps: 1) Initialize w 0 and P 0 ; 2) For k = 0, 1, . . . , a) Feed x k into the network to obtain the output y k ; b) Use back-propagation to compute H k in (29); c) Apply the recursion in (30). The matrix P k is the approximate error covariance matrix, which models the correlations between network parameters and thus delivers second-order derivative information, effectively accelerating the training speed. For example, in [54] it was shown that training an MLP using EKF requires orders of magnitude lower number of epochs than standard back-propagation. In [53], some variants of the EKF training paradigm are discussed. The neural network represented by h k can be a recurrent network, and trained in a similar fashion; the noise covariance matrix R k is scaled to play a role similar to the learning rate adjustment; to reduce computational complexity, a decoupling scheme is employed which divides parameters w k into mutually exclusive groups and turns P k into a block-diagonal matrix. In this section we will explore close connections between neural networks and typical families of signal processing algorithms, that are clearly revealed by unrolling techniques. We also review selected theoretical advances that provide formal analysis and rigorous guarantees of unrolling approaches.

A. Connections to Sparse Coding
The earliest work in establishing the connections between neural networks and sparse coding algorithms dates back to Gregor et al. [11], which we reviewed comprehensively in Section II-B. A closely related work in the dictionary learning literature is the task driven dictionary learning algorithm proposed by Julien et al. [55]. The idea is similar to unrolling: they view a sparse coding algorithm as a trainable system, whose parameters are the dictionary coefficients. This viewpoint is equivalent to unrolling the sparse coding algorithm into a "network" of infinite layers, whose output is a limit point of the sparse coding algorithm. The whole system is trained end-to-end (task driven) using gradient descent, and an analytical formula for the gradient is derived.
Sprechmann et al. [56] propose a framework for training parsimonious models, which summarizes several interesting cases through an encoder-decoder network architecture. For example, a sparse coding algorithm, such as ISTA, can be viewed as an encoder as it maps the input signal into its sparse code. After obtaining the sparse code, the original signal is recovered using the sparse code and the dictionary. This procedure can be viewed as a decoder. By concatenating them together, a network is formed that enables unsupervised learning. They further extend the model to supervised and discriminative learning.
Dong et al. [35] observe that the forward pass of CNN basically executes the same operations of sparse-coding based image super resolution [36]. Specifically, the convolution operation performs patch extraction, and the ReLU operation mimics sparse coding. Nonlinear code mapping is performed in the intermediate layers. Finally, reconstruction is obtained via the final convolution layer. To a certain extent, this connection explains why CNN has tremendous success in single image super-resolution.
Jin et al. [23] observe the architectural similarity between the popular U-net [4] and the unfolded ISTA network. As sparse coding techniques have demonstrated great success in many image reconstruction applications such as CT reconstruction, this connection helps explain why U-net is a powerful tool in these domains, although it was originally motivated under the context of semantic image segmentation.

B. Connections to Kalman Filtering
Another line of research focuses on acceleration of neural network training by identifying its relationship with Extended Kalman Filter (EKF). Singhal and Wu [54] demonstrated that neural network training can be regarded as a nonlinear dynamic system that may be solved by EKF. Simulation studies show that EKF converges much more rapidly than standard back-propagation. Puskorius and Feldkamp [52] propose a decoupled version of EKF for speed-up, and apply this technique to the training of recurrent neural networks. More details on establishing the connection between neural network training and EKF are in the box "Neural Network Training Using Extended Kalman Filter". For a comprehensive review of techniques employing Kalman filters for network training, refer to [53].

C. Connections to Differential Equations and Variational Methods
Differential equations and variational methods are widely applied in numerous signal and image processing problems. Many practical systems of differential equations require numerical methods for their solution, and various iterative algorithms have been developed. Theories around these techniques are extensive and well-grounded, and hence it is interesting to explore the connections between these techniques and modern deep learning methods.
In [14], Chen and Pock adopt the unrolling approach to improve the performance of Perone-Malik anisotropic diffusion [57], a well-known technique for image restoration and edge detection. After generalizing the nonlinear diffusion model to handle non-differentiability, they unroll the iterative discrete partial differential equation solver, and optimize the filter coefficients and regularization parameters through training. The trained system proves to be highly effective in various image reconstruction applications, such as image denoising, single image super-resolution, and JPEG deblocking.
Recently, Chen et al. [58] identify the residual layers inside the well-known ResNet [41] as one iteration of solving a discrete Ordinary Differential Equation (ODE), by employing the explicit Euler method. As the time step decreases and the number of layers increases, the neural network output approximates the solution of the initial value problem represented by the ODE. Based on this finding they replace the residual layer with an ODE solver, and analytically derive associated back-propagation rules that enable supervised learning. In this way, they construct a network of "continuous" depth, and achieve higher parameter efficiency over conventional ResNet. The same technique can be applied to other deep learning techniques such as normalizing flows.

D. Selected Theoretical Studies
Although LISTA successfully achieves higher efficiency than the iterative counterparts, it does not necessarily recover a more accurate sparse code compared to the iterative algorithms, and thorough theoretical analysis of its convergence behavior is yet to be developed.
Xin et al. [59] study the unrolled Iterative Hard Thresholding (IHT) algorithm, which has been widely applied in 0 norm constrained estimation problems and resembles ISTA to a large extent. The unrolled network is capable of recovering sparse signals from dictionaries with coherent columns. Furthermore, they analyze the optimality criteria for the network to recover the sparse code, and verify that the network can achieve linear convergence rate under appropriate training.
In a similar fashion, Chen et al. [20] establish a linear convergence guarantee for the unrolled ISTA network. They also derive a weight coupling scheme similar to [59]. As a follow-up, Liu et al. [19] characterize optimal network parameters analytically by imposing mutual incoherence conditions on the network weights. Analytical derivation of the optimal parameters help reduce the parameter dimensionality to a large extent. Furthermore, they demonstrate that a network with analytic parameters can be as effective as the network trained completely from data. For more details on the theoretical studies around LISTA, refer to the box "Convergence and Optimality Analysis of LISTA".
Papyan et al. [60] interpret CNN as executing finite iterations of the Multi-Layer Convolutional Sparse Coding (ML-CSC) algorithm. In other words, CNN can be viewed as unrolled ML-CSC algorithm. Notably, the convolution operations naturally emerge out of a convolutional sparse representation, while the commonly seen soft-thresholding operation can be regarded as symmetrized ReLU. They also analyze the ML-CSC problem and offer theoretical guarantees such as uniqueness of the multi-layer sparse representation, stability of the solutions under small perturbations, and effectiveness of the ML-CSC algorithm in terms of sparse recovery. In a recent follow-up work [61], they further propose dedicated iterative optimization algorithms for solving the ML-CSC problem, and demonstrate superior reconstruction accuracy and higher efficiency over other conventional algorithms.
Papyan et al. [60] interpret CNNs as executing finite iterations of the Multi-Layer Convolutional Sparse Coding (MLCSC) algorithm. In other words, CNN can be viewed as an unrolled ML-CSC algorithm. In this interpretation, the convolution operations naturally emerge out of a convolutional sparse representation, with the commonly used soft-thresholding operation viewed as a symmetrized ReLU. They also analyze the ML-CSC problem and offer theoretical guarantees such as uniqueness of the multi-layer sparse representation, stability of the solutions under small perturbations, and effectiveness of ML-CSC in terms of sparse recovery. In a recent follow-up work [61], they further propose dedicated iterative optimization algorithms for solving the ML-CSC problem, and demonstrate superior efficiency over other conventional algorithms such as ADMM and FISTA for solving the multi-layer bais purisuit problem.

A. Distilling the Power of Algorithm Unrolling
In recent years, algorithm unrolling has proved highly effective in achieving superior performance and higher efficiency in many practical domains. A question that naturally arises is, why is it so powerful? Fig. 15 provides a high-level illustration of how algorithm unrolling can be advantageous compared with both traditional iterative algorithms and generic neural networks, from a functional approximation perspective. By parameter tuning and customizations, a traditional iterative algorithm spans a relatively small subset of the functions of interest, and thus has limited representation power. Consequently, it is capable of approximating a given target function reasonably well, while still leaving some gaps that undermine performance in practice. Nevertheless, iterative algorithms generalize relatively well in limited training scenarios. From a statistical learning perspective, iterative algorithms correspond to models of high bias but low variance.
On the other hand, a generic neural network is capable of more accurately approximating the target function thanks to its universal approximation capability. Nevertheless, as it typically consists of an enormous number of parameters, it constitutes a large subset in the function space. Therefore, when performing network training the search space becomes large and training is a major challenge. The high dimensionality of parameters also requires an abundant of training samples and generalization becomes an issue. Furthermore, network efficiency may also suffer as the network size increases. Generic neural networks are essentially models of high variance but low bias.
In contrast, the unrolled network, by expanding the capacity of iterative algorithms, can approximate the target function more accurately, while spanning a relatively small subset in the function space. Reduced size of the search space alleviates the burden of training and requirement of large scale training datasets. Since iterative algorithms are carefully developed based on domain knowledge and already provide reasonably accurate approximation of the target function, by extending them and training from real data unrolled networks can often obtain highly accurate approximation of the target functions. As an intermediate state between generic networks and iterative algorithms, unrolled networks typically have relatively low bias and variance simultaneously. Table III summarizes some features of iterative algorithms, generic networks and unrolled networks.

Convergence and Optimality Analysis of LISTA
Although it is shown in [11] that LISTA achieves empirical higher efficiency than ISTA through training, several conceptual issues remain to be addressed. First, LISTA does not exhibit superior performance over ISTA, not even under particular scenarios; second, the convergence rate of LISTA is unknown; third, LISTA actually differs from ISTA by introducing artificial parameter substitutions; and finally, the optimal parameters are learned from data, and it is difficult to have a sense of what they look like.
To address these open issues, several recent theoretical studies have been conducted. A common assumption is that there exists a sparse code x * ∈ R m which approximately satisfies the linear model y ≈ Wx * where W ∈ R n×m and m > n. More specifically, it is commonly assumed that x * 0 ≤ s for some positive integer s, where · 0 counts the number of nonzero entries. Xin et al. [59] examine a closely related sparse coding problem: where k is a pre-determined integer to control the sparsity level of x. In [59] a network is constructed by unrolling the Iterative Hard-Thresholding (IHT) algorithm, which has similar form as ISTA. At layer l, the following iteration is performed: where H k is the hard-thresholding operator which keeps k coefficients of largest magnitude and zeroes out the rest. Xin et al. [59] proved that in order for the IHT-based network to recover x * , it must be the case that for some matrix Γ, which implies that the implicit variable substitution W t = I− 1 µ W T W and W e = 1 µ W T may not play such a big role as it seems, as long as the network is trained properly so that it acts as a generic sparse recovery algorithm. Furthermore, Xin et al. showed that upon some modifications such as using layer-specific parameters, the learned network can recover the sparse code even when W admits correlated columns, a scenario known to be particularly challenging for traditional iterative sparse coding algorithms. Chen et al. [20] perform similar analysis on LISTA with layer-specific parameters, i.e., in layer l the parameters W l t , W l e , λ l are used instead. Similar to Xin et al. [59], they also proved that under certain mild assumptions, whenever LISTA recovers x * , the following weight coupling scheme must be satisfied asymptotically: which shows that the implicit variable substitutions may be inconsequential in an asymptotic sense. Therefore, they adopted the following coupled parameterization scheme: and proved that the resulting network recovers x * in a linear rate, if the parameters W k e , λ k ∞ k=1 are appropriately selected. They further integrate a support selection scheme into the network. The network thus has both weight coupling and support selection structures and is called LISTA-CPSS. Liu et al. [19] extend Chen et al. [20]'s work by analytically characterizing the optimal weights W k e for LISTA-CPSS. They proved that, under certain regularity conditions, a linear convergence rate can be achieved if W k e , λ k k are chosen in a specific form. This implies that the network with analytic parameters can be asymptotically as efficient as the trained version. Although the analytic forms may be nontrivial to compute in practice, their analysis helps reducing the number of network parameters dramatically.

B. Trends: Expanding Application Landscape and Addressing Implementation Concerns
A continuous trend in recent years is to explore more general underlying iterative algorithms. Earlier unrolling approaches were centered around the ISTA algorithm [11], [23], [22], while recently other alternatives have been pursued such as Proximal Splitting [56], ADMM [12], Half Quadratic Splitting [13], to name a few. Consequently, a growing number of unrolling approaches as well as novel unrolled network architectures appear in recent publications.
In addition to expanding the methodology, researchers are broadening the application scenarios of algorithm unrolling. For instance, in communications Samuel et al. [62] propose a deep network called DetNet, based on unrolling the projected gradient descent algorithm for least squares recovery. In multiple-input-multiple-output detection tasks, Det-Net achieves similar performance to a detector based on semidefinite relaxation, while being than 30 times faster. Furthermore, DetNet exhibits promising performance in handling ill-conditioned channels, and is more robust than the approximate message passing based detector as it does not require knowledge of the noise variance. More examples of emerging   unrolling techniques in communications can be found in [63]. The unrolled network can share parameters across all the layers, or carry over layer-specific parameters. In the former case, the networks are typically more parameter efficient. However, how to train the network effectively is a challenge because the networks essentially resemble RNNs and may similarly suffer from gradient explosion and vanishing problems. In the latter case, the networks slightly deviate from the original iterative algorithm and may not completely inherit its Algorithm: Input x 0 , Output x L for l = 1, 2, . . . , L − 1 do 16. An alternative approach to algorithm unrolling is to replace one step of the iterative algorithm with an intact conventional neural network. theoretical benefits such as convergence guarantees. However, the networks can have enhanced representation power and adapt to real world scenarios more accurately. The training may also be much easier compared with RNNs. In recent years, a growing number of unrolling techniques allow the parameters to vary from layer to layer.
An interesting concern relates to deployment of neural networks on resource-constrained platforms, such as digital single-lens reflex cameras and mobile devices. The heavy storage demand renders many top-performing deep networks impractical, while straightforward network compression usually deteriorates its performance significantly. Therefore, in addition to computational efficiency, nowadays researchers are paying increasing attention to the parameter efficiency aspect, and increasing research attention is paid to algorithm unrolling.
Finally, there are other factors to be considered when constructing unrolled networks. In particular, many iterative algorithms when unrolled straightforwardly may introduce highly non-linear and/or non-smooth operations such as hardthresholding. Therefore, it is usually desirable to design algorithms whose iteration procedures are either smooth or can be well approximated by smooth operators. Another aspect relates to the network depth. Although deeper networks offer higher representation power, they are generally harder to train in practice [41]. Indeed, techniques such as stacked pretraining have been frequently employed in existing algorithm unrolling approaches to overcome the training difficulty to some extent. Taking this into account, iterative algorithms with faster convergence rate and simpler iterative procedures are generally considered more often.

C. Alternative Approaches
Besides algorithm unrolling, there are other approaches for characterizing or enhancing interpretability of deep networks. The initial motivation of neural networks is to model the behavior of the human brain. Traditionally neural networks are often interpreted from a neurobiological perspective. However, discrepancies between actual human brain and artificial neural networks have been constantly observed. In recent years, there are other interesting works on identifying and quantifying network interpretability by analyzing the correlations between neuron activations and human perception. One such example is the emerging technique called "network dissection" [64] which studies how the neurons capture semantic objects in the scene and how state-of-the art networks internally represent highlevel visual concepts. Specifically, Zhou et al. [64] analyze the neuron activations on pixel-level annotated datasets, and quantify the network interpretability by correlating the neuron activations with groundtruth annotations. Bau et al. [65] extends this technique to generative adversarial networks. These works complement algorithm unrolling by offering visual and biological interpretations of deep networks. However, they are mainly focused on characterizing the interpretability of existing networks and are less effective at connecting neural networks with traditional iterative algorithms and motivating novel network architectures.
Another closely related technique is to employ a conventional deep network as drop-in replacement of certain procedures in an iterative algorithm. Fig. 16 provides a visual illustration of this technique. The universal approximation theorem [66] justifies the use of neural networks to approximate the algorithmic procedures, as long as they can be represented as continuous mappings. For instance, in [51] Metzler et al. observe that one step of ISTA may be treated as a denoising procedure, and henceforth can be replaced by a denoising CNN. The same approach applies to AMP, an extension to ISTA. In a similar fashion, in [67] Gupta et al. replace the projection procedure in projected gradient descent with a denoising CNN. Shlezinger et al. [68] replace the evaluation of log-likelihood in the Viterbi algorithm with dedicated machine learning methods, including a deep neural network.
This technique has the advantage of inheriting the knowledge about conventional deep networks, such as network architectures, training algorithms, initialization schemes, etc. In addition, in practice this technique can effectively complement the limitations of iterative algorithms. For instance, Shlezinger et al. [68] demonstrated that, by replacing part of the Viterbi algorithm with a neural network, full knowledge about the statistical relationship between channel input and output is no longer necessary. Therefore, the resulting algorithm achieves higher robustness and better performance under model imperfections. Nevertheless, the procedures themselves are still approximated abstractly via conventional neural networks.

VI. CONCLUSIONS
In this article we provide an extensive review of algorithm unrolling, starting with LISTA as a basic example. We then showcased practical applications of unrolling in various realworld signal and image processing problems. In many application domains, the unrolled interpretable deep networks offer state-of-the art performance, and achieve high computational efficiency. From a conceptual standpoint, algorithm unrolling also helps reveal the connections between deep learning and other important categories of approaches, that are widely applied for solving signal and image processing problems.
Although algorithm unrolling is a promising technique to build efficient and interpretable neural networks, and has already achieved success in many domains, it is still evolving. We conclude this article by discussing limitations and open challenges related to algorithm unrolling, and suggest possible directions for future research.
Proper Training of the Unrolled Networks: The unrolling techniques provide a powerful principled framework for constructing interpretable and efficient deep networks; nevertheless, the full potential of unrolled networks can be exploited only when they are trained appropriately. Compared to popular conventional networks (CNNs, auto-encoders), unrolled networks usually exhibit customized structures. Therefore, existing training schemes may not work well. In addition, the unrolled network sometimes delivers shared parameters among different layers, and thus it resembles RNN, which is wellknown to be difficult to train [69]. Therefore, many existing works apply greedy layer-wise pre-training. The development of well-grounded end-to-end training schemes for unrolled networks continues to be a topic of great interest.
A topic of paramount importance is how to initialize the network. While there are well studied methods for initializing conventional networks [70], [2], how to systematically transfer such knowledge to customized unrolled networks remains a challenge. In addition, how to prevent vanishing and exploding gradients during training is another important issue. Developing equivalents or counterparts of established practices such as Batch Normalization [71] and Residual Learning [41] for unrolled networks is a viable research direction.
Bridging the Gap between Theory and Practice: While substantial progress has been achieved towards understanding the network behavior through unrolling, more works need to be done to thoroughly understand its mechanism. Although the effectiveness of some networks on image reconstruction tasks has been explained somehow by drawing parallels to sparse coding algorithms, it is still mysterious why state-of-the art networks perform well on various recognition tasks. Furthermore, unfolding itself is not uniquely defined. For instance, there are multiple ways to choose the underlying iterative algorithms, to decide what parameters become trainable and what parameters to fix, and more. A formal study on how these choices affect convergence and generalizability can provide valuable insights and practical guidance.
Another interesting direction is to develop a theory that provides guidance for practical applications. For instance, it is interesting to perform analysis that guide practical network design choices, such as dimensions of parameters, network depth, etc. It is particularly interesting to identify factors that have high impact on network performance.
Improving the Generalizability: One of the critical limita-tions of common deep networks is its lack of generalizability, i.e., severe performance degradations when operating on datasets significantly different from training. Compared with neural networks, iterative algorithms usually generalize better, and it is interesting to explore how to maintain this property in the unrolled networks. Preliminary investigations have shown improved generalization experimentally of unrolled networks in a few cases [33] but a formal theoretic understanding remains elusive and is highly desirable. From an impact standpoint, this line of research may provide newer additions to approaches for semi-supervised/unsupervised learning, and offer practical benefits when training data is limited or when working with resource-constrained platforms.