{'fraction_non_alphanumeric': 0.03649325109502101, 'fraction_numerical': 0.01191114686689908, 'mean_word_length': 4.610456369107322, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Online Phase Reconstruction via DNN-based Phase Differences Estimation

This paper presents a two-stage online phase reconstruction framework using causal deep neural networks (DNNs). Phase reconstruction is a task of recovering phase of the short-time Fourier transform (STFT) coefficients only from the corresponding magnitude. However, phase is sensitive to waveform shifts and not easy to estimate from the magnitude even with a DNN. To overcome this problem, we propose to use DNNs for estimating differences of phase between adjacent time-frequency bins. We show that convolutional neural networks are suitable for phase difference estimation, according to the theoretical relation between partial derivatives of STFT phase and magnitude. The estimated phase differences are used for reconstructing phase by solving a weighted least squares problem in a frame-by-frame manner. In contrast to existing DNN-based phase reconstruction methods, the proposed framework is causal and does not require any iterative procedure. The experiments showed that the proposed method outperforms existing online methods and a DNN-based method for phase reconstruction.


I. INTRODUCTION
P HASE reconstruction of short-time Fourier transform (STFT) coefficients is important for various audio technologies such as speech enhancement [1]- [6], audio source separation [7]- [11], and text-to-speech synthesis [12]- [15]. As the structure of audio signals is apparent in the magnitude of STFT coefficients, ordinary methods for these technologies have focused on manipulating the magnitude. After obtaining the magnitude, the corresponding phase is required to reconstruct a time-domain signal. Although the phase of an observed signal is available in speech enhancement and audio source separation, it often causes artifacts and residual interference [16], [17]. In text-to-speech synthesis, the magnitude is generated from linguistic features, and thus the phase is fully unavailable. Hence, phase reconstruction of STFT coefficients is helpful for many applications.
As phase reconstruction methods without prior knowledge, consistency-based methods have been widely used [18]- [20]. These methods are based on the relation among nearby STFT coefficients owing to the window and its overlapping nature. To recover this relation, most consistency-based methods are given as iterative optimization algorithms. Meanwhile, PGHI [21] does not require such iterative procedures or the prior knowledge. PGHI is based on a relation between the magnitude and phase of the STFT coefficients [33], [34]; partial derivatives of phase can be analytically calculated from magnitude under some assumptions. This relation enables PGHI to reconstruct phase by integrating phase derivatives. As a result, PGHI has achieved promising results despite lacking any prior knowledge about the target signal.
In contrast, sinusoidal-model-based and DNN-based methods leverage prior knowledge about the target signal. The assumption of sinusoidal-model-based methods is that the target signal consists of sinusoids [22], [23]. This assumption allows one to approximate the phase derivative with respect to time by the frequency of each sinusoid. Then, the phase can be reconstructed by integrating the approximated derivative over time. However, such theoretically derived methods are only applicable to specific signals, and DNN-based methods are promising because DNNs can automatically learn prior knowledge from a training dataset. Since some DNN-based methods have been successfully applied to phase reconstruction [29], we also propose to use DNNs to exploit prior knowledge learned from the training dataset.  While many phase reconstruction methods are offline algorithms, online phase reconstruction is highly desired in a wide range of applications, including incremental text-tospeech [35] and low-latency audio source separation [36]. Therefore, except for the DNN-based methods, the aforementioned methods have been extended to the online setting [22], [23], [37], [38]. A promising method is an extension of PGHI called real-time PGHI (RTPGHI) [38]. Although it outperforms the consistency-based method [37] and the sinusoidal-model-based method [22], there remains room for improvement. First, the STFT phase-magnitude relation is valid only in the continuous setting, and thus estimated phase differences contain some errors in the discrete setting. Second, to approximate the phase derivatives, the centered difference scheme used in PGHI is not allowable in the online setting without look-ahead frames. Instead, RTPGHI uses the backward difference that results in lower performance than the original PGHI. The strong modeling capability of DNNs can improve the estimation accuracy of phase differences.
In this paper, we propose a DNN-based online phase reconstruction framework. As illustrated in Fig. 2, the proposed framework consists of two stages: ( i ) estimating the phase differences from the magnitude and (ii) reconstructing the phase from the estimated phase differences. First, we estimate the phase differences with respect to time (TPD) and frequency (FPD) by using causal DNNs. This DNN-based estimation is expected to be robust to the mismatch of the STFT phasemagnitude relation by leveraging the prior knowledge acquired from a training dataset. Second, we recurrently reconstruct phase from the estimated differences. To handle the phase differences efficiently, we treat them as the ratios of complex STFT coefficients. Then, the phase is reconstructed by solving a weighted least-squares problem of STFT coefficients.
Through several experiments, we confirmed the effectiveness of the proposed two-stage framework compared to existing online phase reconstruction methods.
Note that this paper is related to our conference paper [30], in which we developed the basic concept of the two-stage phase reconstruction framework in the offline setting. In this paper, we extend it to an online method with improvement on all components, i.e., both the first and the second stages. The contributions of this paper are summarized as follows: • proposing an online phase reconstruction framework using causal DNNs, while our previous work [30] focused on the offline setting; • applying convolutional neural networks (CNNs) to estimate phase differences, which is motivated by the STFT phase-magnitude relation; • presenting a novel method for reconstructing phase from its differences by solving the weighted least squares problem of complex STFT coefficients; • investigating and comparing the performance of various online phase reconstruction methods.
The rest of the paper is organized as follows. In Section II, offline and online phase reconstruction problems are formulated. Section III explains the STFT phase-magnitude relation, PGHI, and its online extension, RTPGHI. DNN-based phase reconstruction methods are also reviewed. The proposed twostage framework for DNN-based online phase reconstruction is introduced in Section IV. In Section V, the proposed method is compared with various online phase reconstruction methods, and then the effectiveness of both stages is investigated. Finally, Section VI concludes this paper.
II. PROBLEM FORMULATION STFT 1 of a discrete signal χ with respect to a real symmetric window g of length L is defined as where X[m, n] is the (m, n)th entry of the STFT coefficients, i is the imaginary unit, α is a time shifting step, n = 0, . . . , N − 1 and m = 0, . . . , M − 1 are the time-frame and frequency indices, respectively. These symbols used in this section are summarized in Table I. Let us denote the magnitude and phase of the STFT coefficients X by A and Φ, respectively: where Arg(·) returns the principal value of the complexargument of its input. We consider the task of phase reconstruction that aims at estimating the target phase Φ while allowing the ambiguity of 3  Mapping from the magnitude to the phase Arg(·) Mapping from a complex scalar to its principal argument 2π. That is, our objective is to construct the a mapping F (·) such that: where N ∈ Z M ×N is an arbitrary integer-valued array.
Offline phase reconstruction uses the STFT magnitude at all time-frames as in (4), which can be written as follows: where a n = [A[0, n], . . . , A[M − 1, n]] T , and (·) T denotes the transpose. However, (6) is not applicable to the online setting. In real-time applications, we should estimate the phase at each time-frame only from the magnitudes up to the current timeframe and few look-ahead frames: φ n = F (. . . , a n , . . . , a n+NLA ), where φ n = [ Φ[0, n], . . . , Φ[M − 1, n]] T , and N LA ∈ N is the number of look-ahead frames. The number of time-frames that affect the current output depends on the map F (·). When using a causal CNN [41], it depends on the receptive field of the CNN. Meanwhile, when using a recurrent neural network (RNN), the output at the current time-frame implicitly depends on the magnitudes at all the past time-frames. In this paper, a system is said to be causal if it does not require future information to compute its current output, i.e., N LA = 0.

III. RELATED WORKS
In this section, after explaining PGHI and RTPGHI, we review the DNN-based phase reconstruction methods. The symbols used in this section are listed in Table II.

A. Phase Gradient Heap Integration (PGHI)
PGHI is a non-iterative phase reconstruction method based on the STFT phase-magnitude relation derived from the definition of continuous STFT [21]. In the continuous setting, STFT of a function y ∈ L 2 (R) with respect to a window function h ∈ L 2 (R) is defined as DNN for estimating phase from the given magnitude L(·, ·) Periodic loss function where A and ϕ represent the magnitude and phase, respectively. Let us define the Gaussian window as follows: where σ is a parameter of the Gaussian window. When using the Gaussian window for STFT, both magnitude and phase are partially differentiable with respect to both time and frequency. In addition, the following phase-magnitude relation of STFT can be derived [21]: This relation indicates that the phase derivatives can be analytically calculated from the corresponding log-magnitude. We can thus reconstruct the phase by integrating its gradient up to the global constant phase. PGHI exploits the relations in (10) and (11) to compute the phase gradient in the discrete setting, where STFT is defined as (1). In PGHI, phase gradient is approximated by using the second order centered differences of log-magnitude: where A[m, n] = log(A[m, n]), and β is a constant depending on the window. Note that V c [m, n] and U c [m, n] approximate phase derivatives sampled on the time-frequency (T-F) grid. The phase is reconstructed by numerically integrating the backward phase differences. Since there are multiple possible paths for the integration, PGHI adaptively chooses one of the following four integration paths: where V [m, n] and U [m, n] are approximate backward TPD and FPD, respectively. They are given by averaging the estimated phase gradient in (12) and (13): Note that the oracle backward TPD and FPD are given by respectively. The relation between the phase derivatives and the backward phase differences are illustrated in Fig. 3. While the former is defined for every T-F points, the latter is defined as the relation between adjacent T-F bins.
In the numerical integration, PGHI omits phase at a T-F bin whose magnitude is small because phase differences should be unreliable at such T-F bins. Instead, random phase is assigned to such T-F bins for simplicity. We refer the reader to the paper [21] and codes 2 for more details of implementation.

B. Real-time PGHI (RTPGHI)
RTPGHI is an online extension of PGHI [38]. When allowing one look-ahead frame, i.e., N LA = 1 in (7), the phase gradient can be approximated by the centered difference scheme as in (12) and (13). However, the centered difference in (13) is not applicable to the causal setting (i.e., N LA = 0) because A[m, n + 1] is not accessible. Hence, RTPGHI approximates the phase derivative with respect to the frequency by using the second order backward time-difference of the log-magnitude: Then, the phase is reconstructed via one of the integration paths except (15) because Φ[m, n + 1] is not available. Although RTPGHI achieved promising results, it has some limitations. First, the phase-magnitude relation in (10) and (11) is only valid for the continuous case, and hence the estimated TPD and FPD in (12) and (13) contain errors in the discrete setting. Moreover, this relation assumes the use of the Gaussian window which has infinite support in the time domain. Such a window is not allowed in real-time applications. Second, the experimental results in [38] showed that the second order backward difference approximation in (22) degrades the quality of the reconstructed signals from that of the centered difference in (13).

C. DNN-based Phase Reconstruction
DNN-based phase reconstruction has gained increasing attention because of strong modeling capability of DNNs [24]- [32]. A DNN-based method can handle various signals by learning prior knowledge from a training dataset. A straightforward approach is to model the map F (·) in (4) by a DNN. When training such a DNN, we should consider the periodic nature of the target phase Φ[m, n] because phase is given as a complex-argument. Ordinary loss functions for a regression problem, including the mean squared error, are not suitable for training in such a situation.
To address this issue, several approaches have been presented. One approach uses a DNN to estimate complex STFT coefficients X instead of their phase Φ [24], [28], [29]. Another approach quantizes the target phases and estimates their indices [8], [9]. As a result of recasting the regression problem as a classification problem, the periodic nature of the phase is circumvented. Neither approach directly estimates the phase to avoid dealing with a circular variable.
A periodic loss function has been proposed to train a DNN that directly estimates the continuous circular phase [26], [27]. In this approach, a DNN F θ (·) directly estimates the phase: where θ is a set of parameters of the DNN. To measure the error between the target phase Φ[m, n] and the estimated phase Φ[m, n], a periodic loss function satisfying is considered, where b is an arbitrary integer. For instance, the negative cosine loss function is given by By using the periodic loss function, a DNN for estimating the continuous phase is trained as follows: where we omit the summation over the training dataset because the DNN treats each pair of A and Φ separately. The estimated phase has the ambiguity of 2π due to the use of the periodic loss function. This ambiguity is not a problem 5 Ratio between STFT coefficients at the adjacent frequency-bins Ψn Feature matrix for estimating the phase differences by DNNs Λn Diagonal matrix representing the reliability of estimated TPD Γn Diagonal matrix representing the reliability of estimated FPD Maps W(·) Wrapping operator G θ time (·) DNN for estimating BPD from the given magnitude H θ freq (·) DNN for estimating FPD from the given magnitude P(·) Autoregressive map for reconstructing phase from TPD and FPD Arg(·) Element-wise map from complex scalars to their principal arguments when calculating the complex STFT coefficients with the given magnitude as A[m, n] exp(iΦ[m, n]). The direct phase estimation with a DNN is still hard because a small perturbation of magnitude might imply a large phase difference.

IV. PROPOSED ONLINE PHASE RECONSTRUCTION
In this section, we propose a DNN-based online phase reconstruction framework that consists of two stages. Section IV-A shows the motivation of the two-stage framework. Then, its overview is introduced in Section IV-B. The detail of each stage is explained in Sections IV-C and IV-D, respectively. The weighting rule in the second stage is presented in Section IV-E. The symbols used in this section are summarized in Table III.

A. Motivation: Sensitivity of Phase to Waveform Shift
A DNN-based phase reconstruction method in [26] is formulated as Φ = F θ (A). When training such a DNN in a supervised manner, not only the periodic nature of the phase but also the sensitivity to waveform shifts becomes a problem. Considering the Fourier transform, its phase is sensitive to waveform shifts, while its magnitude is shift-invariant. This is approximately true for STFT when waveform shifts are small. Furthermore, when the sign of a time domain signal is inverted, the corresponding STFT phase is shifted by π without changing magnitude. It is thus difficult to completely determine the phase from given magnitude 3 .
The effects of a waveform shift on STFT magnitude and phase are depicted in Fig. 4. We used an utterance in the LJ speech dataset 4 and that shifted by 0.5 ms. Their TPD and FPD are depicted in the third and fourth rows where we wrap them by using the following wrapping operator: Phase of the shifted signal is noticeably different from the original one even though the magnitude remained almost the same. We thus expect that the phase itself is not easy to 3 STFT magnitude is not completely shift-invariant, and thus the phase can be reconstructed except for the ambiguity of ±π under several assumptions [43], [44]. However, these phase reconstruction methods require excessive computation and are not suitable for the online setting. 4 The LJ speech dataset is available in online: https://keithito.com/ LJ-Speech-Dataset/. The harmonic structure is apparent in the FPD but vague in the TPD. We thus modify TPD to BPD [3] 5 : As depicted in the bottom row of Fig. 4, the harmonic structure is more clear in BPD than in TPD. We thus expect that BPD and FPD are easier to estimate by DNNs, where this expectation will be experimentally confirmed in Section V-H.

B. Overview: Two-stage Online Phase Reconstruction
The example in the previous subsection suggested that directly estimating phase is difficult because a DNN must connect small changes in magnitude to large differences in phase, as illustrated in Fig. 5-( i ). Such an unstable map is not easy to model by a DNN. In contrast, we use DNNs to estimate BPD and FPD as depicted in Fig. 5-(ii). Since a small change in log-magnitude results in small changes in BPD and FPD, the maps from the log-magnitude to the phase differences should be easily modeled by DNNs. After estimating BPD and FPD, the phase is reconstructed in a frame-by-frame manner. This two-stage phase reconstruction is summarized in Fig. 2. At the first stage, causal DNNs G θtime (·) and H θ freq (·) estimate BPD w n ∈ R M and FPD u n ∈ R M −1 at the nth frame, respectively, as follows: w n = G θtime ( a n−NLB , . . . , a n ), (29) u n = H θ freq ( a n−NLB , . . . , a n ), where N LB ∈ N is the number of look-back frames. We stress that both DNNs are causal and do not use magnitude at future time-frames, i.e., N LA = 0. At the second stage, the phase is reconstructed from the estimated phase differences. We design the following map P(·) that computes the phase at the nth time-frame from that at the (n−1)th time-frame with the phase differences and magnitude: φ n = P( φ n−1 , v n , u n , a n , a n−1 ), where v n is the estimated TPD computed from the estimated BPD w n . This map is constructed based on a weighted least squares problem of complex STFT coefficients. Its detail is postponed to Section IV-D. Since both stages do not require information on future time-frames, the proposed framework causally reconstructs the phase in a frame-by-frame manner.
According to the phase-magnitude relation, phase derivatives can be approximated by differences of log-magnitude in the surrounding T-F bins as in (12) and (13). In PGHI, the phase differences are approximated by averaging the phase derivatives at the T-F grids as in (18) and (19). These operations can be implemented by convolution in the T-F domain except for 2παm/M in (12). In the online setting, RTPGHI uses the second order backward difference in (22), which can also be implemented by convolution. While these mathematical formulations are concrete, we expect that estimation accuracy can be improved by exploiting prior knowledge of a target signal. For example, mixed derivative of phase is useful for analyzing harmonic signals [45], and instantaneous frequency of a sinusoidal component can be estimated from its spectral peak [22]. To acquire such complicated phase information from a dataset, DNNs should be effective.
We employ convolution layers that can efficiently aggregate information in the surrounding T-F bins. Our DNNs consist of the mean subtraction and 1-D frequency convolution layers (FreqConv) as in Fig. 6. We concatenate the log-magnitude up to the current time-frame and subtract its mean: Ψ n = N ( a n−NLB , . . . , a n ), where Ψ n ∈ R M ×(NLB+1) is a frame-wise feature, and N (·) subtracts the mean of its inputs. This map just changes the global magnitude within the inputted N LB + 1 frames and retains the STFT phase-magnitude relation. The following first FreqConv layer treats temporal adjacencies of the inputted T-F bins as channels. As a result, the FreqConv layer can perform a causal convolution along the time-frame and mimic the operations in (13) and (22). In detail, the number of channels of the FreqConv layer corresponds to the kernel size of the causal convolutions along the time-frame. The frame-wise feature Ψ n is passed to multiple FreqConv layers. We combine the FreqConv layers with the gating mechanism [46] as follows: where the two FreqConv layers have different parameters. This mechanism can adaptively control the information passed to the next layer. Its effectiveness has been confirmed in DNNbased phase reconstruction [29]. In this first stage of the proposed framework, each feature Ψ n is handled separately, and thus it is causal. CNNs have difficulty of using the absolute T-F location due to their translation invariance. In the phase-magnitude relation in (12), absolute frequency information 2παm/M is required to compute the phase derivative with respect to time. It is thus difficult to estimate TPD by a CNN. In contrast, BPD removes the absolute frequency information in (28) and is expected to be easily estimated by a CNN.
Supervised learning is straightforward for training DNNs that estimate BPD and FPD, because the pairs of magnitude and phase differences are easily calculated from time-domain signals. The target phase differences should be treated as circular variables because they inherit the periodic nature of the phase. Hence, we measure the errors of the estimated phase differences by the periodic loss functions as follows:

D. Second Stage: Online Phase Reconstruction From Phase Differences
As in (31), the second stage of the proposed framework P(·) recurrently estimates the phase based on the estimated phase differences. The phase differences estimated by DNNs have ambiguity of 2π due to the use of the periodic loss function. The second stage of the proposed framework must take care of this ambiguity. Since it is not easy to directly handle such phase differences with the ambiguity, we propose to convert them to the ratios of complex STFT coefficients as follows: where V [m, n] = W [m, n] + 2παm/M . Note that the oracle versions of these ratios are given by which cannot be computed because the phase of X[m, n] is not available. This conversion is depicted in Fig. 7. The main advantage of this conversion is that the 2π ambiguity of the estimated phase differences is avoided. On the basis of the complex ratios, we formulate an optimization problem and estimate the phase by solving it. Let us consider the complex ratios at the nth time-frame v n and u n given by To enforce the complex ratios between successive time-frames close to v n , we minimize the following function T (·) with respect to an optimization variable z n ∈ C M : where ] T is the estimated STFT coefficients at the previous time-frame, diag(·) returns the diagonal matrix whose diagonal elements are its input vector, and z 2 Λn = z H Λ n z. Meanwhile, to enforce the complex ratios between adjacent frequencies close to u n , the following function S (·) is also minimized: where the matrix D n ∈ R M −1×M is defined as and the other entries are zero. The weights Λ n in (42) and Γ n in (43) are diagonal matrices that reflect the reliability of the estimated TPD and FPD, respectively. The detail of the weights is explained in the next subsection. By using these two functions, the map P(·) in (31) is realized as follows: where ( (42) for the next time-frame. The optimization problem in (47) aims to estimate complex STFT coefficients that are consistent with the ratios calculated from the phase differences. It can be solved in a closed form: 8 where the mth entry of y n is given by v[m, n] X[m, n − 1]. By using (48), the proposed method reconstructs the phase in a frame-by-frame manner without any iterative optimization.

E. Weighting Rule and Initialization
In the optimization-based phase reconstruction given in (46) and (47), the weights, Λ n and Γ n , are important to improve the quality of the reconstructed signal. The phase differences with large weights are maintained, and thus the weights must be designed based on the reliability of the estimated phase differences. We propose to design the mth diagonal entry of Λ n and Γ n by using the given magnitude: where p is a parameter for compressing or enhancing the magnitudes, and γ 0 ≥ 0 is a parameter to balance the two weights. These weights are based on the assumption that the ratios of complex STFT coefficients are accurate when magnitude at the related T-F bins is large. As a special case, the proposed method in (48) results in the integration of the estimated TPD over time when γ 0 = 0: This phase reconstruction was already used in a DNN-based method [25]. Its performance is limited because the relation between adjacent STFT coefficients in the frequency direction is neglected. The proposed method with γ 0 > 0 uses the relations in both time and frequency directions. Another related work [47] applies some weight designed from the given magnitude to training of DNNs that estimate phase. In contrast, we use the weights for reconstructing phase from the phase differences but not for training DNNs. The recurrent phase reconstruction in (48) is not applicable to the initial time-frame because x n−1 is not given. We compute the phase at the initial time-frame Φ[m, 0] by accumulating the estimated FPD. In our preliminary experiments, however, it often resulted in a similar performance with other initialization methods, e.g., the zero and random phases. This should be because the estimated FPD is unreliable due to a small magnitude at the initial time-frame. The proposed method is robust against errors at the T-F bins with small magnitudes because the relations between the T-F bins with large magnitudes are emphasized by the weights in (49)- (50). In detail, if the T-F bins at the previous time-frame have small magnitude, the proposed method tries to maintain the estimated FPD at the current time-frame and neglect the phase at the previous time-frame.

V. EXPERIMENTS
In this section, we investigate the performance of the proposed DNN-based two-stage framework in online phase reconstruction. The experimental conditions are described in Section V-A. Section V-B compares the proposed framework with various online and offline phase reconstruction methods. The generalization capability and robustness of the proposed framework are shown in Sections V-C and V-D, respectively. The effectiveness of our CNN for the first stage is validated in Section V-E. We investigate the effect of the weight parameters, p and γ 0 , on the quality of the reconstructed signals in Section V-F. Section V-G demonstrates the effectiveness of the optimization-based phase reconstruction method in the second stage. Finally, the proposed two-stage framework is compared with direct phase reconstruction in Section V-H.

1) Dataset and STFT Parameters:
Evaluations were performed on the LJ speech dataset that consists of 13100 audio clips uttered by a female speaker. The audio clips were sampled at 22050 Hz and randomly splitted into three subsets: 12500 clips for training, 300 clips for validation, and 300 clips for testing as in [48]. During the training, the utterances were further divided into about 1-second-long segments (24064 samples). The validation set was used to optimize the hyperparameters for the second stage. STFT was computed with the Hann window, where the window size and shift size were 1024 and 256 samples, respectively. We used ltfatpy 6 to implement STFT and related transformations.
2) DNN Configuration and Training Setup: The DNN used in the following experiments is illustrated in Fig. 6. It consists of the mean subtraction layer, a FreqConv layer, and five FreqGatedConv layers followed by another FreqConv layer. We set the number of look-back frames N LB to 3 based on the overlap of the window for STFT. Other configurations are summarized in Table IV. To train the DNNs, we used the RAdam optimizer [50] for 100 epoch where the batch size was 32. We linearly warmed up the learning rate for 5 epochs to 0.0004 and adopt the halfperiod cosine scheduler [51]. We applied a weight decay of 10 −6 and a gradient clipping of 10 for stable training, which were implemented in Pytorch [52]. 9 Fig. 8. Boxplots of PESQ, ESTOI, and LSC for 300 reconstructed utterances. Blue and red boxes correspond to online and offline phase reconstruction methods, respectively. Higher PESQ and ESTOI indicate better sound quality. Lower LSC indicates better phase reconstruction.

3) Evaluation Metrics:
The results of phase reconstruction were evaluated by three objective measures. The first one is the log-spectral convergence (LSC) [53] defined by where X[m, n] = A[m, n] e Φ[m,n] , and · Fro denotes the Frobenius norm. When the estimated phase is perfect, i.e., Φ = Φ, iSTFT and STFT do not alter the magnitude of X, and LSC becomes −∞. The second and third ones are the wide-band extension of the perceptual evaluation of subjective quality (PESQ) [54] and the extended short-time objective intelligibility (ESTOI) [55]. These objective measures have been commonly used to evaluate naturalness and intelligibility of the results of phase-aware speech enhancement [56] and separation [57].

B. Comparison to Existing Online Phase Reconstruction
To validate the effectiveness of the proposed DNN-based phase reconstruction, we compared the proposed method with three online phase reconstruction methods: RTPGHI [38], a consistency-based phase reconstruction method called realtime iterative spectrogram inversion (RTISI) [37], and a sinusoidal-model-based phase reconstruction method called single pass spectrogram inversion (SPSI) [22]. For RTPGHI and RTISI, we set the number of look-ahead frames N LA to 0, i.e., all methods were set causal. We also investigated the performance of two offline methods: PGHI and an offline version of the proposed method. In the offline proposed method, we additionally concatenated three look-ahead frames as the input of the DNNs. All existing methods are implemented in PHASERET [42]. In RTISI, the number of pertime-frame iterations was set to 5, which is the default value of PHASERET. We would like to stress that this is not a fair comparison because only the proposed method uses a DNN trained on utterances of the target speaker. This comparison, however, demonstrates the great potential of incorporating the prior knowledge into online phase reconstruction. The generalization capability of the proposed method is validated in the next subsection. Furthermore, the performance of the existing methods combined with the DNNs are investigated in Section V-G. PESQ, ESTOI, and LSC of the reconstructed signals are summarized in Fig. 8. The two methods that do not utilize prior knowledge of target signals, RTPGHI and RTISI, resulted in similar performance. Although SPSI considers the sinusoidal model for target signals, it performed worse than the other methods in our experiment. This should be a consequence of the mismatch between the signal model and actual signals. The proposed method was able to outperform all of the existing methods. Note that the proposed method also outperformed the offline PGHI. This result confirms the advantage of leveraging prior knowledge of the target signals learned by DNNs. The offline proposed method substantially improved the performance from that of the online version. That is, the performance of the two-stage phase reconstruction can be improved by leveraging look-ahead frames.

C. Generalization for Unseen Speakers
To clarify the generalization capability of the proposed method, we evaluated it on unseen speakers. While the training and validation data were from the LJ speech dataset, the evaluation was performed on utterances of three females (p225, p228, p229) and three males (p226, p227, p232) from the VCTK corpus. For each speaker, 100 utterances were randomly selected and resampled at 22050 Hz as in [29].
The experimental results are summarized in Table V. Even on the unseen speakers, the proposed method outperformed RTPGHI which was the best reference method in the previous experiment. This result confirms the generalization capability of the proposed framework even when the training dataset consists of utterances of a single speaker. Training on utterances of multiple speakers might improve the generalization capability further.

D. Application to Mel-Spectrogram Inversion
In many applications, the given STFT magnitude contains some errors. To investigate the robustness against such errors, we validated RTPGHI and the proposed method on the magnitude recovered from that compressed to the mel scale. Mel- spectrograms have been widely used as acoustic features in audio synthesis, and phase reconstruction has been applied to the magnitudes recovered from them [58], [59]. In this experiment, the power-compressed magnitude in the linear scale with 513 bins, A[m, n] 0.3 , was converted to the mel scale with a smaller number of bins M mel ∈ {80, 160, 240, 320, 400}. Then, the mel-scale magnitude is converted back to that in the linear scale with regular power by solving a nonnegative least squares problem, which is implemented in Librosa [60]. The power compression has been used to maintain the components with a small magnitude in least squares as in [61] and was effective for both RTPGHI and the proposed method. The smaller M mel caused more error in the recovered linear-scale magnitude.
PESQ and ESTOI of the reconstructed signals are shown in Fig. 9. In addition to RTPGHI and the proposed method, we evaluated the recovered magnitude with the true phase. This is an upper bound of the performance of phase reconstruction. When M mel > 80, the proposed method substantially outperformed RTPGHI. Recently, DNNs have been used to recover the magnitude in the linear scale from that in the mel scale [58], [59]. These DNN-based methods should improve the quality of the recovered magnitude from the nonnegative least squares used in this experiment. We thus expect that the performance of the proposed method is improved by incorporating it with the DNN-based estimation of the magnitude in the linear scale.

E. Effectiveness of CNN to Estimate BPD and FPD
To validate the effectiveness of the proposed CNN for estimating BPD and FPD, we compared its estimation accuracy with that of an FCN. The CNN was the same as that used in the previous experiment (Fig. 6). The FCN comprised 3 gated linear units of 1024 units and a linear output layer as in [26], where its number of total parameters was 8929k which is about 43 times more than that of the CNN. The input of the FCN was the log-magnitudes up to the current time-frame as in (32), but we concatenated them along with the frequency direction. The training configuration was the same as in Section V-A2. The The histograms of AWE of the estimated BPD and FPD are illustrated in Fig. 10. These histograms are more biased towards the left when the estimates were more accurate. AWE of RTPGHI is summarized in the rightmost column, where TPD computed by (18) was converted to BPD. The accuracy of FPD was significantly worse than that of BPD because RTPGHI in the causal setting must use the second order backward difference in (22) instead of the centered difference for computing FPD. If the second order centered difference was used by allowing one look-ahead frame, the median of AWE was reduced to 0.389 from 0.701. Even though the DNNs did not use any look-ahead frames, they achieved notably better accuracy compared to RTPGHI. By efficiently aggregating information in the surrounding T-F bins, the CNN outperformed the FCN with 43 times fewer parameters. Consequently, as shown in Table VI, the objective measures of the reconstructed utterances were significantly improved by using the CNN for the estimation of phase differences.
To demonstrate the difficulty of estimating TPD using a CNN, we compared a CNN and an FCN by directly estimating TPD. Note that the proposed framework does not estimate TPD itself, and hence we trained another DNN for this experiment. The histograms of AWE for TPD estimation are 11 12. Average PESQ and ESTOI on the validation set. The parameters varied on the logarithmic scale in both axes. We limit the color ranges to clarify the peaks, which results in saturation for p > 10 0.4 . depicted in Fig. 11. The FCN achieved performance similar to that for BPD in Fig. 10. In contrast, TPD estimation by CNN resulted in much more error compared to that of BPD. This result indicates that estimation of TPD is difficult for the CNN as discussed in Section IV-C. Hence, the conversion of the target from TPD to BPD is essential for the CNN.
F. Effect of the Weight Parameters p and γ 0 As discussed in Section IV-E, the weights are important in the second stage of the proposed framework. In our experiments, the optimal weight parameters were obtained by using the validation set, where the phase differences were estimated by the CNNs trained as Section V-A. The search range of p and γ 0 were set to [0.1, 10] and [1, 100], respectively. Fig. 12 shows PESQ and ESTOI of the reconstructed signals on the validation set. The proposed framework performed well with wide range of parameters, i.e., it is not so sensitive to these parameters. ESTOI took the maximum value 0.996 at p = 10 −0.4 and γ 0 = 10, which also gives a high PESQ value. Hence, these parameters were used in the other experiments. Fig. 13. Boxplots of PESQ, ESTOI, and LSC of signals reconstructed from the phase differences approximated by (18) and (19). The time integration of TPD [25] and the adaptive integration of TPD and FPD [38] are abbreviated as Time Int. and Adaptive Int., respectively.

G. Evaluation of Various Methods for Reconstructing Phase From Estimated Phase Differences
In this experiment, the second stage of the proposed framework was evaluated. We compared the second stage of the proposed framework with existing methods: the time integration of the estimated TPD used in [25] defined by (51), and the recurrent phase unwrapping (RPU) presented in our conference paper [30]. The difference between RPU and the proposal of this paper is the definition of least squares problems; RPU solves the least squares problem of phase without weighting, while the second stage of the proposed framework solves the least squares problem of complex STFT coefficients with weighting. We also evaluated the adaptive integration of TPD and FPD used in RTPGHI [38]. Note that, if the oracle phase differences are available, all methods can perfectly reconstruct the phase up to the global constant. To investigate their robustness to the error in the phase differences, this experiment used the analytic formulas in (18) and (19) or the CNNs to estimate the phase differences. Fig. 13 summarizes the results using the phase differences computed by the analytic formulas in (18) and (19). The performance of the adaptive integration and the proposed method was significantly better than that of the time integration and RPU. The former group uses the magnitude to reconstruct the phase from the estimated phase differences, while the latter group does not. As a result, the former group is more robust to the estimation error at T-F bins with small magnitudes. Fig. 14 shows the results using the phase differences estimated by the CNNs. According to Section V-E, the estimation accuracy of FPD was improved by using the CNN from that of (19). The performance was improved in all methods except for the time integration that does not use FPD. The proposed optimization-based method outperformed the other methods including the adaptive integration. This should be because the proposed method jointly optimizes all the phase at each timeframe based on the carefully designed weight.

H. Comparison with DNN-based Direct Phase Reconstruction
In this experiment, the proposed two-stage framework is compared with the direct phase reconstruction using the FCN or CNN. For the direct phase reconstruction, the DNN was trained to minimize the negative cosine loss function of phase in (25). We further refined the estimated phase by an iterative offline phase reconstruction method called Griffin-Lim algorithm (GLA) as in the original paper [26]. The number of iterations of GLA was 100. The reconstructed phase was evaluated by AWE of the phase differences 7 . We also investigated the total performance of the proposed twostage framework in this way. To be specific, we evaluated not the phase differences estimated in the first stage but those 7 We did not evaluate the estimated phase directly because global phase shift is not reflected in the perceptual quality.
This evaluation is illustrated in Fig. 15.
The results of the DNN-based phase reconstruction methods using FCNs and CNNs are summarized in Figs. 16 and 17, respectively. The direct phase reconstruction resulted in the lowest performance regardless of the types of DNN 8 . This result indicates the difficulty of directly estimating phase as discussed in Section IV-A. In contrast, the two-stage framework with the FCNs successfully reconstructed phase up to global constant phase. Although the direct phase reconstruction performed better when GLA was applied, the proposed framework with the CNNs outperformed it. Note that this comparison is unfair because GLA is an iterative offline method that uses the information from all time-frames. Even though the proposed two-stage method is non-iterative and causal, it was able to outperform the combination of the DNN-based and iterative methods.
Although the proposed two-stage method performed better than the existing methods, there is room for improvement. According to Fig. 10, AWE of BPD and FPD estimated in the first stage using the FCNs were 0.417 and 0.480, respectively, and those using the CNNs were 0.091 and 0.117, respectively. The results in Figs. 16 and 17 indicates that the output of the second stage was worse than those obtained in the first stage in terms of phase differences. This implies that the second stage of the proposed framework still has room for improvement. Refinement of the second stage is left as a future work.

VI. CONCLUSION
In this paper, we have presented a two-stage online phase reconstruction framework. In the framework, BPD and FPD are estimated by the causal DNNs based on 1-D frequency convolution layers. Then, phase is reconstructed from the estimated phase differences by analytically solving the weighted least squares problem of complex STFT coefficients in a frame-byframe manner. We confirmed that the proposed method outperformed existing online phase reconstruction methods. We also demonstrated the effectiveness of the two-stage framework by comparison with the direct reconstruction method.
In future works, we will apply the two-stage online framework to low-latency speech enhancement and separation. In these applications, the noisy phase of an observed signal is useful to improve the estimation accuracy of the phase differences. Fine-tuning of the DNNs used in the first stage to maximize the quality of the signals reconstructed by the second stage is also a possible direction for improving the proposed framework. Together with improvement of the second stage as discussed in Section V-H, optimizing the whole process of the proposed framework is the next step for realizing a better phase reconstruction method.