{'fraction_non_alphanumeric': 0.03956603513922657, 'fraction_numerical': 0.014541951465663562, 'mean_word_length': 5.04869589345172, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LAPNet: Non-rigid Registration derived in k-space for Magnetic Resonance Imaging

Physiological motion, such as cardiac and respiratory motion, during Magnetic Resonance (MR) image acquisition can cause image artifacts. Motion correction techniques have been proposed to compensate for these types of motion during thoracic scans, relying on accurate motion estimation from undersampled motion-resolved reconstruction. A particular interest and challenge lie in the derivation of reliable non-rigid motion fields from the undersampled motion-resolved data. Motion estimation is usually formulated in image space via diffusion, parametric-spline, or optical flow methods. However, image-based registration can be impaired by remaining aliasing artifacts due to the undersampled motion-resolved reconstruction. In this work, we describe a formalism to perform non-rigid registration directly in the sampled Fourier space, i.e. k-space. We propose a deep-learning based approach to perform fast and accurate non-rigid registration from the undersampled k-space data. The basic working principle originates from the Local All-Pass (LAP) technique, a recently introduced optical flow-based registration. The proposed LAPNet is compared against traditional and deep learning image-based registrations and tested on fully-sampled and highly-accelerated (with two undersampling strategies) 3D respiratory motion-resolved MR images in a cohort of 40 patients with suspected liver or lung metastases and 25 healthy subjects. The proposed LAPNet provided consistent and superior performance to image-based approaches throughout different sampling trajectories and acceleration factors.


I. INTRODUCTION
agnetic resonance imaging (MRI) is a valuable and versatile tool in clinical diagnostics. Its capability of assessing anatomy and functional processes within the human body in a non-invasive manner makes it an essential imaging modality. However, MRI is prone to several artifacts which can deteriorate images significantly up to the point of non-diagnostic quality. Due to the long acquisition times in MRI, motion is one of the major extrinsic factors influencing image quality. Motion patterns can be categorized into rigid motion such as global translations or rotations of stiff structures T.K. and S.G. are with Medical Image And Data Analysis (MIDAS.lab), Department of Diagnostic and Interventional Radiology, Tübingen, Germany (e-mail: {thomas.kuestner | sergios.gatidis}@med.uni-tuebingen.de).
T.K, G.C. R.B. and C.P are with the School of Biomedical Engineering and Imaging Sciences, King which arises from movements of whole body parts and nonrigid motion including (local) deformations under an affine model which mainly occurs in the thorax and abdominal region caused by physiological motion. Other body parts, such as the bowel, can be affected as well. Patient and physiological motion induces ghosting along the phase-encoding direction and/or blurring of the image content. The manifestation in the image mainly depends on the imaging sequence and sampling trajectory. Motion visualization, estimation and correction are thus important tasks when reconstructing or processing MRI data. Several prospective and retrospective motion compensation approaches have been developed to minimize or correct for motion induced artifacts. These include fast imaging sequences [1,2] to enable acquisitions within resting periods (e.g. under breath-holds) or minimal motion (e.g. mid-diastole); tracking of motion by sensors (MR navigators [3][4][5][6][7][8][9][10], cameras [11], respiratory belts or electrocardiogram [12]) to limit data acquisition to periods with minimal movement (e.g. diastole, end-expiration); application of motion-robust acquisition schemes [13]; prospectively corrected acquisitions [14]; retrospective motion-corrected reconstruction [15,16] and motion-resolved imaging [17,18].
In relation to motion-corrected approaches, one can differentiate between the correction of rigid motion (e.g. head motion) and non-rigid periodic motion (e.g. respiratory and cardiac motion). Rigid motion can be tracked by MRI or other external sensors and modelled in k-space for translational motion as linear phase drifts that can be incorporated into the acquisition (prospective correction) or reconstruction (retrospective correction) schemes [8][9][10]13]. Non-rigid motion can be tracked by MRI or sensors as well, but on the other hand is more challenging to correct as it involves local deformations in image space which are related to changes in the entirety of kspace (acquisition space) in a non-trivial way. Correction of non-rigid motion is therefore usually performed retrospectively on motion-resolved images [19]. Accordingly, a prior image reconstruction step is required. M This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMI.2021 To achieve fast acquisitions, motion-resolved thoracic and abdominal images are usually highly undersampled in k-space. In order to reconstruct aliasing-free images these methods rely on reconstruction schemes that for example exploit sparsity or low-rank redundancies in the spatial and/or motion directions to solve the ill-posed reconstruction problem [2,20]. These approaches require careful parametrization and fine-tuning between regularization and data consistency to avoid residual aliasing (under-regularized) and staircasing or blurring artifacts (over-regularized).
After reconstruction, motion fields can be estimated in image space from the motion-resolved reconstructed images. Spatiotemporal redundancies can be exploited in the reconstruction to improve image quality [21][22][23][24][25][26][27] which implicitly perform a motion correction without directly relying on motion fields. Alternatively, motion-compensated reconstructions can be carried out which iterate between an image reconstruction and image registration step [28][29][30][31][32][33][34][35]. While motion-compensated methods have shown to efficiently utilize motion information, they require a significantly increased computational demand and the achievable imaging acceleration is limited by the quality of the motion-resolved undersampled reconstruction.
Non-rigid motion estimation is usually formulated in image space using diffusion-based [36], parametric spline-based [37] or optical flow-based registration methods [38]. In general, motion estimation can be guided by external motion surrogate signals [28,39], initial motion field estimates [29,30], from motion-aliased images [31] or low-frequency image contents, derived from central k-space data [40,41]. It was shown in MR-MOTUS [41] that conventional image registrations can be modelled as a low-rank optimization for undersampled cases of non-rigid respiratory motion and rigid head motion..
Recently, deep-learning based approaches have been presented [42] to learn generalizable rigid [43,44] and nonrigid [45][46][47][48][49][50] registrations in image space for medical images. FlowNet [51] and FlowNet-2 [52] have been proposed as supervised optical flow registration networks for fully-sampled 2D natural scene images and which has also been translated to registration of MR images [53,54]. In CarMEN [55] motion fields were derived in an encoder-decoder convolutional neural network from dynamic multi-slice 2D MR image stacks to perform cardiac motion estimation.
To summarize the above, motion correction methods depend on i) reliable motion-resolved images or low-frequency kspaces (i.e. low-resolution images) from which motion fields can be estimated, ii) require good initial motion field estimates, iii) rely on other external motion sensor signals, iv) constraint the imaging or v) motion field (e.g. local affine) optimization. In case of highly undersampled data, aliasing or blurring artifacts in the reconstructed images can impair the registration process as reconstruction errors can propagate into the image registration and/or low-resolution images may not provide sufficient information for accurate registration. Aliasing-free registration from accelerated acquisitions can be of use for i) integration into motion-compensated reconstructions, ii) interor intra-modality motion correction of other imaging acquisitions (e.g. PET/MR motion correction), or iii) investigation of subject-depending motion behaviour (e.g. cardiac wall motion).
In this work, we propose a deep-learning based approach to perform fast and accurate non-rigid motion estimation directly from acquired k-space based on optical flow equations. This work focuses on "image" registration carried out in k-space. The working principle originates from our recently introduced Local All-Pass (LAP) technique [56][57][58], an image-based 3D non-rigid optical flow registration. We will first describe the basic concepts of LAP and illustrate its extension to a k-space based 3D non-rigid registration [59]. We will then introduce the proposed deep-learning non-rigid registration network, named LAPNet, operating on motion-resolved k-space data to derive 3D deformation fields. The proposed LAPNet is compared against traditional image-based registrations (LAP and NiftyReg [60]) and FlowNet-S [51] operating on image inputs. We investigate the proposed approach in 40 patients with suspected liver or lung metastases and 25 healthy subjects for retrospectively and prospectively undersampled data of 3D respiratory motion-resolved MR imaging.

II. THEORY
In 3D non-rigid image registration, one tries to estimate a motion field/deformation field/motion model = , , which maps all voxels of a moving image to a reference/fixed image by minimizing some dissimilarity metric = arg min ( , ( )). (1) To solve this optimization task, one can utilize parametric (e.g. B-splines, thin-plate splines) [61] or non-parametric (e.g. optical flow, diffusion) [62] approaches. In the following, we will derive the notation for an efficient optical flow-based nonrigid registration in image space (II.A), which builds upon [56][57][58], from which we extend it to a non-rigid registration in kspace (II.B). A computationally efficient implementation is then achieved by deep learning registration network, named LAPNet (II.C).

A. Local All-Pass (LAP) in image space
Under the assumption of local brightness consistency, i.e. the intensity remains constant when flowing from the moving to the fixed image, the volumetric images can be linked by a translation. The optical flow can be stated as = − ( ) for deforming a moving image ∈ ℝ to a fixed image ∈ ℝ with a translation ( ) ∈ ℝ at voxel position = [ , , ] inside the field-of-view (FOV) determined by the 3D spatial size , and . This relationship also holds true for deforming local neighborhoods ≪ FOV with ∈ . Under the hypothesis of a motion flow continuum, one can approximate any global non-rigid flow ∈ ℝ existing on a large grid by a large sum of many local translational flows ( ) in a smaller support , This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. In other words, any non-rigid deformation can be regarded as a sum of local translational displacements if 3D motion flow is smoothly varying.
For a translational flow, one can equivalently state the occurring translation in Fourier domain, following the shifting property of the Fourier transform resulting in a multiplication of the moving k-space ∈ ℂ with a linear phase to obtain the fixed k-space ∈ ℂ at all k-space locations = , , . Accordingly, if one then defines = , the fixed k-space becomes an all-pass (i.e. = 1) filtered version with in Fourier domain of the moving kspace . Assuming ideal sampling with a sinc kernel, we have a digital version of the all-pass filter with (2π,2π,2π) periodic frequency response.
Putting together equations (3) to (5), one sees that solving for the non-rigid motion field in (1), i.e. performing a non-rigid registration (in image space), can be equivalently explained by finding the appropriate local all-pass filters in Fourier space and deriving the flow from it.
To estimate the all-pass filter , we use the key algorithmic idea proposed in [63], that the all-pass filtering between two images can be expressed linearly in terms of forward and backward filtering − , yielding which is all-pass by design, i.e. = ( ) / (− ) = 1. Consequently equation (4) respectively (2) becomes with all-pass ℎ , forward and backward − filters in image space. The discrete filters are approximated by a filter basis series where denotes the number of optimal filter coefficients without loss of generality setting =1. As pointed out in [64], a canonical basis with support in of cubic size × × would act as an upper bound for the possible deformation of 2 ⁄ − 1. Hence, following the analysis in [65], an all-pass filter can be approximated by a finite filter basis , if the basis spans the derivatives of an isotropic function.
The size of the filter basis confines the non-rigid motion estimation to estimating translational flows in a local region, respectively estimating local all-pass filters, given by the filter support and defining then the LAP equation Combining (8) and (9) allows to conclude the non-rigid LAP registration problem in image space for which at each image position ∈ ℝ the optimal filter coefficients are estimated by minimizing the dissimilarity (e.g. mean-squared-error, MSE) between and . For sake of simplicity, describes the window function indicator of the neighborhood in (3). The local all-pass filter yields the local translational flow at the central voxel in and can be derived from (6) as Sliding the window ( ) over all voxel positions provides the non-rigid flow as stated in (3).
In order to deal with motion of varying strength, a multiresolution approach is applied in which the size of is decreased per multi-resolution step, i.e. coarse-to-fine estimation.

B. Non-rigid registration in k-space
Following the key idea of LAP that any non-rigid deformation can be regarded as local translational displacements and together with the Fourier shift property, a k-space representation is obtained in (4) linking a moving k-space = ℱ ( ) to a fixed k-space = ℱ ( ) at all kspace locations , where ℱ describes the Fourier transform.
In order to carry out the estimation of local translations in Fourier domain (i.e. k-space), we need to consider the local windowing which corresponds in k-space to the convolution by a phasemodulated (for various positions) tapering function .Consequently, transforming (10) in Fourier domain yields the non-rigid k-space registration based on the LAP optical flow concept. However, not for all dissimilarity in image space exists a representable counterpart in Fourier domain. Following the Parseval equation we restrict ourselves to being the MSE.
Please also note that in the k-space version, summation is required over all positions and for shifted tapering supports at This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMI.2021.3096131 all positions. Hence, carrying out a registration over the complete spectral support of the FOV ∈ ℝ × × requires operations at each iteration which can be computationally demanding. In comparison, the image-based version only operates on the smaller window ≪ FOV, yielding operations. It should be noted that for undersampled acquisitions, only the acquired k-space locations need to be visited in the registration. Hence for an undersampled acquisition with an acceleration factor of , we obtain a computational reduction of which reduces computational burden only slightly.
We therefore seek to simplify these operations and significantly reduce the computational burden by learning an appropriate registration network that can carry out the previously described non-rigid registration in k-space and which is furthermore not prone to aliasing artifacts arising from undersampled acquisitions, i.e. generalizes for different acceleration strategies.

C. LAPNet: Non-rigid registration network in k-space
First, we will have a closer look at the effect of the tapering function . In image domain, the windowing nulls the image outside of the chosen filter support and allows thus to crop the image to the smaller cubic size. A similar operation can be achieved in k-space by tapering with followed by a regridding to yield an input cube of size × × with ≪ , , . This minimizes the required input dimensions of the network and thereby the memory footprint and number of trainable parameters in the network.
The proposed deep-learning architecture, denoted as LAPNet, for non-rigid registration in k-space is depicted in Fig.  1. The zero-filled 3D moving and fixed k-space is convolved with the tapering function following (12) and subsequently regridded to an empirically optimized size of 33x33x33.k-Spaces are sliced along one spatial dimension and real and imaginary parts of the moving and fixed k-space are concatenated along the channel direction. This bundle of tapered k-space patches of size 33 2 x4 is then passed through a succession of convolutional filters with dyadic increase in kernels and leaky ReLU activation function. Kernel sizes are depicted in Fig. 1. In the last layer a fully connected regression is performed on the average pooled feature map to estimate the in-plane deformations , at the given central location of the k-space input patch for the -th run. To obtain a 3D deformation field = [ , , ], the registration is performed on two orthogonal spatial directions. In the first run ( = 1), kspaces are sliced along the last spatial dimension (readout direction), yielding , . In the second run, slicing can be done e.g. along the first dimension for processing this bundle of k-space patches yielding , . Results are then merged with the previous run to yield = , = 0.5( + ), = at the central voxel location of the k-space cube . The whole non-rigid deformation field (in image domain) is obtained by estimating the deformations , , at all voxel locations by processing tapered k-spaces patches sequentially, i.e. sliding the window over all voxels (as indicated by the arrows in Fig. 1). Flows are estimated from each k-space patch and correspond to the voxel location of the patch center. Flows are in image domain, due to the last fully-connected layer learning a weighted transformation.
This non-rigid registration follows the principle optimization stated in (13) where the network learns the canonical basis of (all-pass) filters ( Fig. 1 convolution filters) as expressed in (8) for a local neighborhood (Fig. 1 tapering) as described in (12). The = 6 consecutive convolutional filters can be regarded as learnable coefficients and basis functions . Training is performed in a supervised manner to optical-flow derived diffeomorphic reference flows of (11). The estimated flows by LAPNet can thus be assumed to be diffeomorphic as well. Moreover, performing flow estimation on a k-space input patch follows the idea of approximating a global non-rigid flow by local translational deformations.
The network is trained in a supervised manner on pairs of moving and fixed k-space inputs with the corresponding reference motion field derived from the image-based LAP [56,57] was employed as the training loss. The network resulted in ~25 million trainable parameters and was trained by an Adam optimizer [66] ( = 0.9, = 0.999) for an initial learning rate of 2.5 ⋅ 10 with learning rate scheduler (division by 2 every 50000 iterations) and a batch size of 64 over 50 epochs on a Nvidia Titan RTX GPU. Training and test data are further specified in section III.B.
The source code is publicly available under MIT license: github.com/lab-midas/lapnet

III. METHODS
A. In-vivo 4D MR acquisition 3D motion-resolved k-space data was obtained in a cohort of 40 patients (60 ± 9 years, 22 female) with suspected liver or lung metastases [58] and 25 healthy subjects (31 ± 4 years, 10 female) [18]. The study was approved by the local ethics committee and all subjects gave written consent. Imaging was performed on a 3T PET/MR (Biograph mMR, Siemens Healthcare, Erlangen, Germany) equipped with a phased array body and spine coil. A 3D T1 weighted spoiled gradient echo sequence was acquired in coronal orientation with a continuous variable-density Poisson Disc undersampling [67] for a time of acquisition (TA) of 5 min. The remaining imaging parameters were TE = 1.23ms, TR = 2.60ms, bandwidth = 890Hz/px and a flip angle of 7°. A matrix size of × × = 256 x 256 x 144 (RO x PE x 3D ⇔ FH x LR x AP) was acquired covering a field-of-view of 500 x 500 x 360 mm 3 . A 2D MR selfnavigation signal (256 x 8 x 1, RO x PE x 3D) was acquired each 200 ms serving as gating signal [18]. MR data were retrospectively gated into = 8 respiratory bins, ranging from end-expiratory to end-inspiratory position, with a Gaussian view-sharing amongst neighbouring bins. Coil sensitivity maps were determined by ESPIRiT [68] from the time-averaged (across all respiratory gates) central fully sampled k-space data. Images were reconstructed by a FOCal Underdetermined System Solver (FOCUSS) [67] producing motion-resolved and complex-valued target images ∈ ℂ respectively target k-spaces = ℱ ∈ ℂ .

B. Training and test data
Training data was derived from reconstructed motionresolved target data / . Reference flows in training were composed of real, smooth and augmented flows. Motion fields were obtained with the image-based LAP registration for = 4 over 5 multi-resolution levels with window sizes per level of = 64, 32, 16, 8, 4 between endexpiratory fixed = ( = 1) and remaining moving bins = ( > 1) of the target image (denoted as real as described realistic motion) with ∈ [1, ] being the bin number. Moreover, arbitrary smoothly varying flows/motion fields were generated with a maximum displacement of 10 voxels drawn from multivariate Gaussian distribution [56,57] (denoted as smooth). Flows were randomly augmented by smoothing (convolution with 5x5x5 Gaussian low-pass), translating (-10 to 10 voxels in each direction), rotating (-25° to 25° in all planes) and by multiplication with arbitrary smoothly varying flows (denoted as augmented).
One objective of the proposed approach is to provide nonrigid registration from undersampled datasets without the need of image reconstruction, i.e. not being limited by aliasing artifacts. In training, the fixed and moving k-spaces are retrospectively undersampled with a sampling operator . Two undersampling strategies ∈ ℝ are employed: 1) a variable-density Poisson-Disc undersampling (vdPD), reflecting a typical incoherent undersampling for compressed sensing like reconstructions and 2) taking a fully-sampled elliptical central region (center), reflecting a low-resolution acquisition. Acceleration factors were randomly distributed in the range of = 1 to 30 where = 1 corresponds to the fullysampled case, i.e. = ( = 1) with ( ) = 1 ∀ . LAPNet was trained jointly on the different undersampling strategies and acceleration factors.
For training, the fixed input corresponds to the fully-sampled ( = 1) or vdPD/center ( > 1) undersampled k-space = ( = 1) of the target acquisition in end-expiratory position and the moving input relates to the registered fullysampled or vdPD/center undersampled k-space = ℱ ∓ , = 1 deformed with the forward motion model (real, smooth or augmented) and bilinear interpolation.
Training data was created by taking k-space patches together with their corresponding flows at random spatial locations in coronal and sagittal orientation from 33 patients and 18 healthy subjects. A total of 15 million training samples with different types of motion (real, smooth, augmented) and undersampling (fully-sampled, vdPD, center) were generated approximately every 5th epoch with empirically determined ratios of 40% real, 20% smooth and 40% augmented. For training a stride of 1 was performed for the Window .
For testing, k-space patches of 7 patients and 7 healthy subjects (not included in training) were taken and flow estimation was conducted at every 2 nd k-space location for LAPNet, i.e. a sliding window with a stride of 2 was performed to boost registration performance during inference, which showed best performance without loss of accuracy. In the fullysampled case ( = 1) fixed and moving k-space are extracted from . In testing for > 1, the continuous sampling in the acquisition is cropped to a shorter scan time duration, and thus mimicking a prospective undersampling. Undersampled fixed and moving k-space inputs for testing were taken after respiratory motion binning of this prospectively accelerated acquisition.

C. Experiments
For comparison, two conventional image-based 3D non-rigid registrations were performed by the image-based LAP (denoted as imageLAP) [56][57][58] which is an optical flow method and by NiftyReg [60] which used a free-form deformation cubic Bspline model [69]. Please note that for = 1, the imageLAP provides the reference flow used in training the networks. Comparison to imageLAP is therefore only performed for > 1. Conventional methods serve as comparison to investigate This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMI.2021.3096131 impact of image acceleration on image-based and non-data adaptive registrations. Both conventional methods were run on an Intel Xeon E5-2697 CPU. Experiments on the conventional non-rigid registration in k-space can be found in [59].
For further comparison, an image-based FlowNet-S [52] network was trained on the respective fully-sampled ( = 1) or vdPD/center ( > 1) undersampled zero-filled images and corresponding flows . Full 2D coronal and sagittal images are used as input. The network was trained by an Adam optimizer ( = 0.9, = 0.999) for an initial learning rate of 10 with learning rate scheduler (division by 2 every 5000 iterations) and a batch size of 16 over 50 epochs to minimize the sEPE loss in (14). Remaining hyperparameters were optimized but found to coincide with published parameters [51]. Training database composition differed to LAPNet in the sense that the best combination of undersampling (fullysampled, vdPD, center) was empirically optimized to yield the smallest possible validation loss. Two instances of FlowNet-S were hence trained for i) fully-sampled + vdPD and ii) fullysampled + center. If not stated otherwise, the best performing case is reported.
The two different undersampling strategies enable to infer which k-space locations can potentially contribute to the nonrigid registration, i.e. is it beneficial to include high-frequency components (vdPD) or is it better to only focus on the lowfrequency range (center). The latter undersampling case thereby also allows a comparison to previously published methods [40,41] operating on low-resolution images (obtained from central k-space sampling) or central k-space data. Testing was performed considering the respective undersampling strategies (fully-sampled, vdPD, center) independently, for LAPNet and FlowNet-S.

D. Evaluations
The end-point error EPE = − and end-angulation error EAE = arg( , ) between the estimated motion field was compared with the reference motion field obtained from imageLAP ( = 1) of the target acquisition over the whole FOV. Structural similarity index (SSIM) [70], normalized root mean squared error (NRMSE) = 1/ √MSE, peak signal-to-noise ratio (PSNR) = 10 log 1/|MSE| and normalized cross-correlation (NCC) were calculated between the deformed moving image = − , > 1 and the end-expiratory fixed target = ( = 1). Bilinear interpolation was performed for non-integer grid points. All quantitative results are reported as mean ± one standard deviation over all voxel positions, gates and test subjects. Due to the lack of a gold-standard ground-truth motion, quantitative results can only be interpreted in relation to each other. Statistical significance was determined with a paired Welch's ttest and Bonferroni correction under the null hypothesis of equal means for unequal variances. P-values < 0.05 were considered statistically significant.

IV. RESULTS
The proposed LAPNet showed statistically significant superior performance in the quantitative analysis of EPE and EAE over changing acceleration factors for vdPD and center sampling as depicted in Fig. 2 and Supplementary Table I. LAPNet has the lowest EPE and EAE amongst all cases and shows a consistent performance throughout acceleration factors. Deviations amongst subjects in conventional imagebased methods (imageLAP, NiftyReg) are larger indicating a less consistent performance. Center sampling shows slightly higher EPE and EAE metrics towards higher accelerations than vdPD indicating that high-frequency information is beneficial for the registration task (both in k-space and image space). The proposed k-space based LAPNet is less affected by this information loss and generalizes better.
Quantitative similarities between the registered images and the fixed images by a voxel-wise intensity comparison is shown as violin distributions in Fig. 6 over all test subjects and accelerations. Percentage differences and statistical pairwise testing to the proposed LAPNet are summarized in Supplementary Table I. The proposed LAPNet outperformed all other methods in each respective metric with statistical significance. LAPNet showed a more consistent performance with lower standard deviations. In center sampling slightly improved performance of image-based methods were observed. The respiratory motion-resolved images of a representative patient with a neuroendocrine tumor in the liver are depicted in Fig. 4 for a fully-sampled acquisition and for prospectively undersampled vdPD and center acquisitions with 8x and 30x acceleration. Deformation fields were obtained by the proposed LAPNet, FlowNet-S, imageLAP and NiftyReg. The LAPNet (i.e. k-space non-rigid registration) shows good performance over all acceleration factors with close resemblance to the target motion field. Motion flows are only estimated in the body trunk with LAPNet whereas estimation in the static (noisy) background is relatively low or smoothed out in comparison to the conventional methods (imageLAP and NiftyReg).
Learning-based approaches (LAPNet and FlowNet-S) better generalize to the changing input data (remaining aliasing and noise impact) than the conventional methods. The data-adaptive learning allows to generalize a network-based registration to a certain extent for different acceleration factors, i.e. a network is able to see through the aliasing and noise artifacts. However, one can appreciate the performance difference between LAPNet (k-space registration) and FlowNet-S (image-space registration). The LAPNet is less obstructed by aliasing artifacts, resulting in smoother and more consistent flows amongst accelerations with closer agreement to the actual underlying respiratory motion which is also reflected in this Fig. 4: Respiratory non-rigid motion estimation in a patient with neuroendocrine tumor in the liver (pointed out by red arrow) by the proposed LAPNet in k-space in comparison to image-based non-rigid registration by FlowNet-S (deep learning), image-based LAP (imageLAP; optical flow) and NiftyReg (cubic B-Splines). Estimated flow displacement are depicted in coronal and sagittal orientation. Reference flows are obtained from imageLAP on fully-sampled images. Undersampling was performed prospectively with a vdPD and center undersampling for 8x and 30x acceleration. Super-inferior liver dome displacement between end-expiratory and end-inspiratory of 1.5 cm was observed. This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMI.2021.3096131 subject in a statistically significant lower ( < 0.001) EPE of 64% ± 3% and lower EAE of 42% ± 6% in LAPNet than FlowNet-S. It can also be appreciated that for higher accelerations the aliasing is less pronounced in the images. Images are blurrier and smoothed out which arises from the fact that vdPD sampling is concentrated towards the low-frequency central region similar to the center sampling.
LAPNet produced consistent results over all acceleration factors as well as for different sampling strategies. Image-based methods (FlowNet-S, imageLAP and NiftyReg) show a similar trend for reducing performance with increasing acceleration factors. FlowNet-S performs better than conventional methods but shows substantial misregistration at the diaphragm. Misregistered flows in the image background were increased for FlowNet-S with center and vdPD sampling even for carefully optimized training database composition. FlowNet-S handles low-resolution image input (center) statistically significant better ( < 0.001) than undersampling artifactaffected images (vdPD) for accelerations ≤ 15. Registration of conventional image-based methods (imageLAP and NiftyReg) failed for center with > 10, yielding random flows. Overall letting us conclude that high-frequency samples, i.e. image edges, carry relevant information for the image-based registrations which are inevitable lost at one point for a decreasing sampled central region size (increasing acceleration ), respectively stronger low-pass filtered image.
In the coronal orientation, superior-inferior respiratory motion is mainly dominating. Static non-moving regions such as the spine are not deformed by the proposed LAPNet, as well as the image-based registrations for the fully-sampled case, indicating that local non-rigid deformations can be correctly captured. In sagittal orientation, the superior-inferior respiratory motion shows an additional posterior displacement. Fig. 5 shows the deformed images obtained from non-rigid registration in undersampled data for the proposed LAPNet and FlowNet-S in a patient with pancreatic carcinoma and liver metastasis. For increasing acceleration factors, a rising trend of misregistered local deformations and non-smooth flows were obtained with the image-based methods yielding overall a less consistent performance. The proposed LAPNet provided consistently good flow estimation over all motion states (endexpiratory to end-inspiratory) and accelerations. Deformed images of LAPNet matched the fixed reference image better than that of FlowNet-S. In general, performance of FlowNet-S was inferior to LAPNet. Fig. 6 and 7 show the motion estimation capability for periodic (Fig. 6) and linear drifting (Fig. 7) respiration in two patients with neuroendocrine tumors. Motion estimates obtained from shorter 1 min portions of the scan (i.e. corresponding to ~25x undersampling), are consistent for cyclic motion. Deep learning based registrations are superior to conventional image registration (NiftyReg). Flows of the proposed LAPNet are in accordance with reference flows. For linear drifting respiration, FlowNet-S and NiftyReg fail to capture deformations accurately. LAPNet produced in this scenario reliable results.
The depicted cases in Fig. 4-7 show the observed performance variations amongst test subjects. No outlier or failed cases were observed in the test cohort with similar performance amongst subjects.
Non-rigid registration with LAPNet required ~12 hours in training whereas inference required only ~18s. FlowNet-S required ~2.5 hours of training and ~12s in inference. This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMI.2021.3096131 Registrations with conventional methods were performed on average in 23 ± 4s for imageLAP and 83 ± 28s for NiftyReg.

V. DISCUSSION
In this work, we propose a deep learning network based nonrigid registration in k-space. The proposed LAPNet was compared against image-based non-rigid registration methods in healthy subjects and patients for fully-sampled acquisitions and two undersampling strategies (vdPD and center). A throughout consistent and superior performance of the proposed LAPNet was found. Non-rigid motion estimation is feasible in k-space which generalizes well for highly accelerated cases. Data-driven learning of image registration (LAPNet, FlowNet-S) outperformed conventional image registrations (imageLAP, NiftyReg) for accelerated acquisitions, although being trained in a supervised manner on imageLAP derived flows. The LAPNet, i.e. k-space registration, showed a more consistent and improved performance over image-based registration (FlowNet-S) for different acceleration factors. and NiftyReg (cubic B-Splines). Motion is estimated from 1 min parts of the scan, corresponding to a ~25x undersampling. Reference flows depict the imageLAP registration from the complete 5 min scan. Cyclic respiration with 1 cm super-inferior displacement was observed. Fig. 7: Respiratory non-rigid motion estimation in a patient with neuroendocrine tumor and liver metastastis (pointed out by red arrow). Motion displacement is estimated by the proposed LAPNet in k-space in comparison to image-based non-rigid registration by FlowNet-S (deep learning) and NiftyReg (cubic B-Splines). Motion is estimated from 1 min parts of the scan, corresponding to a ~25x undersampling. Reference flows depict the imageLAP registration from the complete 5 min scan. Linear drifting respiration with ~3 cm super-inferior displacement was observed. This article has been accepted for publication in IEEE Transactions on Medical Imaging. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TMI.2021.3096131 A comparison of the proposed LAPNet to image registration methods that are more robust towards undersampled aliasing and blurring would be desirable. However, registration methods in this field are primarily paired up or interleaved into an image reconstruction [28][29][30][31][32][33][34][35]. Examination of sole impact and effect of image registration becomes hereby difficult. We therefore used the image-based conventional and deep learning image registrations as comparison. The influence of the receptive field for LAPNet and FlowNet-S were investigated. In LAPNet, an empirically optimized size was determined which provided a good trade-off between computational time and accuracy. For increasing receptive field size, non-rigid motion estimation was less accurate and overall performance dropped. If receptive field was chosen too small, the local neighbourhood was not contributing to the non-rigid motion estimation resulting in non-smooth flows. A good tradeoff was found for = 33 and filter kernel sizes. In contrast to imageLAP, maximum possible displacement that can be registered with LAPNet was not confined by cubic size . Here, the question for maximum possible displacement that can be registered corresponds to maximum possible perceived phase changes. Furthermore, it is expected that data-driven deep learning registrations can benefit from learning various motion patterns and displacement amplitudes. In the examined cases, motion of varying amplitude was seen for which LAPNet provided good registrations. Future investigations are required to examine registration accuracy limits.
We observed that for the same acceleration factor the LAPNet seems to handle motion estimation in the presence of aliasing artifacts (image space) better as obtained with vdPD sampling than from low-resolution data as in center sampling (see Fig. 1, 2). For high acceleration factors (>15x) it seems to be beneficial to include some high-frequency samples in the registration. In these accelerations, more low-frequency samples are contributing to the estimation, suggesting that there might be an optimal amount and distribution of low-frequency and highfrequency samples which contribute to the overall registration. It is hence conceivable training a network to identify the most significant k-space samples contributing to the registration which is related to works of adaptive sampling for image reconstruction [71].
Qualitatively, smoother flow estimations were observed from center sampling in the test cohort. Especially FlowNet-S was operating better under blurring than aliasing artifacts. While quantitative metrics are often used for driving the conventional image registration tasks, the voxel-wise measures do not necessarily reflect the correction of motion [72], but provide a good indicator of how well each of the respective methods perform in relation to each other.
Spatial alignment between flows and undersampled data was achieved by retrospective undersampling in training. In order to account in the supervised training for changing and subjectspecific respiratory motion patterns, augmentation of real flows was performed. Bilinear interpolation can introduce interpolation errors but was found to be sufficient and least computationally demanding for training. Testing was conducted on prospectively undersampled data with varying motion patterns. Consistent and superior performance of the proposed LAPNet was found for cases ranging from mild (up to 1.5 cm liver dome displacement) to strong motion (up to 6 cm) with linear drifting respiration. However, future investigation on larger cohorts is warranted with strongly varying motion patterns.
We acknowledge further limitations of this work. The proposed LAPNet was trained and tested for a single imaging sequence with two different sampling strategies in a coilcombined setting for 2D k-space input. In the future, we want to extend LAPNet for multi-coil 3D k-space input and investigate its generalizability for different imaging sequences and sampling trajectories, e.g. radial sampling, as well as applications. An empirically optimized input size was set, but a multi-resolution approach is conceivable and needs to be investigated in the future. The trained filters are not restricted to have all-pass characteristic but approximate their behaviour which needs to be further investigated in the future. A supervised learning was used which may be biased by the image-based LAP reference flow, potentially also limiting FlowNet-S capabilities. Future work will investigate on selfsupervised and unsupervised learning together with other suitable dissimilarity measures in Fourier domain.

VI. CONCLUSION
A deep-learning based non-rigid registration method, LAPNet, which can be directly performed in the acquired kspace domain is proposed. Results indicated improved performance of LAPNet in comparison to image-based registration approaches for high acceleration factors. LAPNet showed consistent performance throughout different sampling trajectories and acceleration factors. It thus enables non-rigid motion registration from highly accelerated acquisitions.