{'fraction_non_alphanumeric': 0.03550177174567092, 'fraction_numerical': 0.01597914020191215, 'mean_word_length': 4.9402303415409055, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generation

We propose EdgeFormer -- a parameter-efficient Transformer of the encoder-decoder architecture for on-device seq2seq generation, which is customized under strict computation and memory constraints. EdgeFormer proposes two novel principles for cost-effective parameterization and further enhance the model with efficient layer adaptation. We conduct extensive experiments on two practical on-device seq2seq tasks: Machine Translation and Grammatical Error Correction, and show that EdgeFormer can effectively outperform previous parameter-efficient Transformer baselines and achieve very competitive results with knowledge distillation under both the computation and memory constraints. Moreover, we release the pretrained EdgeFormer -- the first publicly available pretrained model that can be easily fine-tuned for English seq2seq tasks with strong results, largely facilitating on-device seq2seq generation in practice.


Introduction
On-device machine learning has drawn increasing attention owing to the unique advantages (Dhar et al., 2019), like real-time processing (without network latency regardless network connection), better privacy (without sending data to the cloud), and lower serving cost (without hosting massive cloud servers). On the other hand, the resource-constrained edge devices prevent many neural networks performing well in the on-device setting. In Natural Language Processing (NLP), ondevice sequence-to-sequence (seq2seq) generation remains challenging, especially for the powerful Transformer (Vaswani et al., 2017) under strict resource constraints in both computation and memory. To customize the Transformer for seq2seq tasks in the on-device setting, we propose EDGEFORMER -a novel parameter-efficient Transformer of the encoder-decoder architecture. EDGEFORMER is structurally similar to the standard Transformer with a deep encoder and shallow decoder, but with a minor modification that it uses an interleaved decoder with shared lightweight feed-forward networks, (a) (b) Figure 2: (a) Encoder layers with shared weights (the same color): the tied weights undermine the specialities of encoder layers to process their specific inputs; (b) Layer adaptation (LA) leverages a small number of layer-specific parameters to help adapt layers with tied weights to fit their specific roles well. as shown in Figure 1. The modified decoder architecture allows EDGEFORMER to apply two novel principles that we propose for cost-effective parameterization: 1) encoder-favored parameterization that suggests we parameterize the encoder using as many parameters as possible; 2) load-balanced parameterization that suggests we balance the load of model parameters to avoid them being either underused or overused in a neural network with shared parameterization.
In addition to novel cost-effective parameterization, EDGEFORMER proposes and applies layer adaptation to further improve the model with tied layers, as Figure 2 shows. Inspired by parameter-efficient task transfer, we investigate efficient layer adaptation approaches that improve the performance with negligible cost.
We conduct experiments in two most practical on-device seq2seq tasks to evaluate EDGEFORMER: machine translation (MT) and grammatical error correction (GEC). We show that EDGEFORMER (with fewer than 10 million model parameters) largely outperforms baseline models in the on-device setting and achieves very competitive performance with knowledge distillation. The int8-quantized EDGEFORMER can perform high-quality on-device MT and GEC (20-30 input tokens on average) within around 100ms latency using two mid-to-high-end CPU cores and less than 50MB RAM.
The main contributions of this paper are three-fold: • We propose EDGEFORMER, a novel parameter-efficient Transformer that achieves the stateof-the-art result in the on-device seq2seq generation setting under strict computing and memory resource constraints. We release the pretrained EDGEFORMER which is the first publicly available pretrained model that can be easily fine-tuned for English seq2seq tasks with strong results, largely facilitating on-device seq2seq generation in practice.
• We propose two novel principles of cost-effective parameterization for EDGEFORMER, largely improving conventional cross-layer parameter sharing methods for Transformers given a tight parameterization budget.
• We propose layer adaptation to help adapt layers with shared weights, and investigate efficient layer adaptation approaches that can further improve the model with little cost.

Architecture
The Transformer is the most popular seq2seq neural network, which follows the encoder-decoder architecture based on multi-head attention mechanism. The Transformer encoder consists of a stack of encoder layers, each of which has a self-attention module followed 2 by a point-wise fully connected feed-forward network:

Self-attention Module
The self-attention module in the Transformer employs the multi-head attention mechanism in which each head maps a query and a set of key-values pairs to an output. The output in a single head is computed as a weighted sum of values according to the attention score computed by a function of the query with the corresponding key. These single-head outputs are then concatenated and again projected, resulting in the final values: where Q, K and V correspond to projected Q, K and V in head i . W Q , W K , W V and W O are linear transformations for queries, keys, values and outputs. As a common practice, all their sizes are d × d. Note that d is the model dimension, while d k in Eq (3) denotes the dimension of queries and keys in a single head. In self-attention modules, Q = K = V .
Feed-forward Network The feed-forward network (FFN) in the original Transformer consists of two linear transformations with a ReLU activation in between: d ffn is usually set to be much larger than d and it is common that d ffn = 4d.
The Transformer decoder consists of a stack of decoder layers. A decoder layer's architecture is similar to an encoder layer but with two exceptions: 1) the self-attention module in a decoder layer needs to mask out the attention from future tokens to prevent leftward information flow; 2) a decoder layer introduces a cross attention module between self-attention and FFN for collecting information from the encoder's final output h. As self-attention, the cross attention module employs the multi-head attention mechanism except that its K = V = h.
In summary, we understand that the main parameters in an encoder layer i are: For a decoder layer j, its main parameters are: are parameters for cross-attention modules. |Φ dj | = 8d 2 + 2d × d decffn .

Parameterization: Full vs Shared
Full Parameterization Full parameterization is a common parameterization approach for Transformer, meaning that each model parameter (excluding embedding) is exclusive and independent without being shared by multiple modules in the network. In a forward pass, each parameter is used only once. Full parameterization allows parameters to be flexible to fit their roles well during model training.
Shared Parameterization Despite the advantages of full parameterization, it is criticized to use large numbers of parameters inefficiently, motivating shared parameterization where multiple modules in a network share parameters. For a model with shared parameterization (e.g., ALBERT), each model parameter (excluding embedding) is exploited more than once in a forward pass. The high parameter efficiency of shared parameterization results in a better result than full parameterization given the same parameterization budget, despite slowing down inference. On the other hand, given a fixed architecture (i.e., the same network depth and width), shared parameterization usually underperforms full parameterization because it has much fewer free model parameters.

On-device Constraints
Computation On-device computer vision (CV) models tend to use 1G FLOPS (0.5G MACS) as a constraint, which is also followed by some work for on-device translation, theoretically allowing peak performance of 50 sentences of 30 tokens per second in a typical mobile CPU of 4 cores (Wu et al., 2020). In our work, however, we propose to relax the FLOPS constraint for seq2seq tasks. One reason is that seq2seq tasks such as MT are heavier by nature than classification tasks and thus it is unreasonable to require a seq2seq model to perform as fast as classification models. As a result, it makes little sense to deploy a seq2seq model on an edge device with too weak computing units. Also, the requirement of on-device seq2seq generation for real-time performance is not so rigid as CV tasks since it is uncommon for an on-device seq2seq model to handle too many concurrent requests in practice; instead, it usually performs online inference or over a small number of input sentences on a local device. Therefore, we propose to use 2G FLOPS (1G MACS) as the constraint. The relaxed constraint helps improve the prediction quality that strongly correlates with user experience for seq2seq tasks, but still ensure the CPU on edge devices to process tens of sentences per second, which is more than sufficient for an on-device seq2seq model. In addition to FLOPS that is a theoretical hardware-independent measurement for computational cost, we also require the runtime latency for an input sentence (typically 20 ∼ 30 tokens on average) to be within around 100ms using two mid-to-high-end CPU cores.
Memory Following previous work (Wu et al., 2020), we limit the number of model parameters (excluding 3 word embedding lookup table) up to 10 million. For runtime memory, we require the memory cost to be less than 50MB.

Architecture
An on-device seq2seq model needs to meet the computation and memory requirement discussed in Section 2.2; furthermore, we expect the model to achieve as good results as possible. For a standard Transformer (Vaswani et al., 2017), we observe the model of 512 dimension can basically meet the FLOPS requirement (2G FLOPS), as shown in Table 1; and its variant (Kasai et al., 2020) with a deep encoder and a shallow decoder allows the model to decode faster in practice at test time despite slightly more FLOPS.
The biggest challenge for an on-device seq2seq model is regarding the model size and memory cost. The number of parameters of a full-parameterized Transformer-base model is about 45 million (excluding the embedding parameters), which is far beyond the parameterization budget (10 million) and unavoidably leads to massive memory cost.
EDGEFORMER is proposed to address the challenge. Instead of disruptive architectural changes as previous research (Wu et al., 2020;Mehta et al., 2020;Panahi et al., 2021), EDGEFORMER's architecture basically follows the standard Transformer consisting of a 12-layer encoder and 2-layer 4 decoder with d = 512 since we find it sufficient to achieve good performance in the on-device setting as long as it can be well parameterized, only with a minor modification that EDGEFORMER proposes to use an interleaved decoder where attention modules are interleaved with shared lightweight FFNs (d decffn < d; in this work, d decffn = d/4) in each decoder layer, as illustrated in Figure 1. The advantages of the decoder's architecture are two-fold, which are both keys for cost-effective parameterization (Section 3.2): • The interleaved structure makes the architecture of encoder and decoder layers consistent (Ma et al., 2021), facilitating shared parameterization of attention modules throughout the encoder and decoder. • As shown in Table 1, the lightweight FFNs that interleave attention modules in the decoder reduce FLOPS and save a large number of parameters for decoder FFNs' parameterization that is very uneconomical.

Cost-effective Parameterization
Due to the tight parameterization budget (i.e., 10 million), EDGEFORMER cannot be fully parameterized as in the standard way; instead, it has to adopt shared parameterization.
As a popular solution to shared parameterization, UNIVERSAL TRANSFORMER proposes to let all its encoder layers share one group of encoder layer parameters and all its decoder layers share one group of decoder layer parameters: and N are the number of encoder and decoder layers respectively.
Although UNIVERSAL TRANSFORMER can save parameters, its parameterization is still not costeffective for EDGEFORMER to perform well for two reasons: First, UNIVERSAL TRANSFORMER uses (over) half of total parameters to parameterize the decoder, which is uneconomical. As shown in Figure 3a, given a fixed architecture (6+6 Transformer, d = 512), densely parameterizing the decoder results in much less performance gain than parameterizing the encoder. This suggests we use as many parameters as possible to parameterize the encoder for the performance.
Second, UNIVERSAL TRANSFORMER's parameterization does not consider the load balance of model parameters. For EDGEFORMER or the Transformers with a deep encoder and shallow decoder, UNIVERSAL TRANSFORMER's parameterization method will overburden parameters in the encoder but underutilize parameters in the decoder. For example, for a 12+2 UNIVERSAL TRANSFORMER, a parameter in the encoder is used 12 times, while a parameter in the decoder is used only twice in a forward pass. As shown in Figure 3b, moderately reusing parameters (e.g., when x ≤ 4) helps better utilize the parameters, resulting in significant performance gain without increasing parameters. However, as the shared parameters are overused (when x > 6), the performance gain will become marginal, which is intuitive because a parameter's capability is limited. This suggests we should try to balance the load of parameters to avoid them being either overused or underused.
Based on the above insights, we parameterize EDGEFORMER in the following two novel principles for cost-effective parameterization: Encoder-favored Parameterization For EDGEFORMER, we parameterize its encoder using as many parameters as possible: except a small number of parameters (d 2 /2) for all lightweight FFNs in the decoder, we use almost all parameters in our budget to parameterize the encoder. In this way, our parameterization budget allows 4 groups of attention parameters (4 × 4d 2 ), 2 groups of encoder FFN parameters (2 × 8d 2 ) and 1 group for decoder lightweight FFNs (i.e., d 2 /2). We do not assign parameters exclusively for attention modules in the decoder; instead, we let them reuse (i.e., share) parameters with the attention modules in the encoder since attention modules in both the encoder and decoder work in the same mechanism and can be effectively shared (Dong et al., 2019). Thanks to the interleaved decoder architecture that makes the structure of encoder and decoder layers consistent, we let the self-attention module in a decoder layer share parameters with its corresponding odd layer in the encoder, and let its cross-attention module share with the corresponding even layer in the encoder, inspired by Ma et al. (2021): Load-balanced Parameterization We try parameterizing EDGEFORMER with a balanced load for each model parameter so that each parameter could be equally exploited in a forward pass. To be specific, the 2 groups of encoder FFN parameters are equally shared by all encoder layers; the 1 group of decoder FFN parameters is shared by light FFNs in the decoder; and 4 groups of attention parameters are shared throughout the encoder and decoder. Except for parameters in the encoder FFNs that are used 6 times, other parameters are all used 4 times in a forward pass. In this way, we avoid parameters being underused or overused, resulting in cost-effective parameterization. Based on the principle, we parameterize EDGEFORMER in the cycle-shared manner (Takase & Kiyono, 2021): (1 ≤ j ≤ 2)

Layer Adaptation
Shared parameterization causes layers with tied weights to become less specialized, as discussed in Section 1. To allow layers with tied weights to be better adapted to their corresponding roles, we propose layer adaptation to further enhance EDGEFORMER. Inspired by parameter-efficient approaches for task transfer, we mainly investigate three efficient layer adaption approaches for EDGEFORMER: Figure 4: (a) Adapter-LA uses a layer-specific LoRA block with rank r < d (dashed square); (b) Prompt-LA uses L layer-specific tokens as the prompt (dotted square) to adapt the mth layer.
Bias-based Layer Adaptation (Bias-LA) Inspired by BitFit (Ben Zaken et al., 2021) that finetunes with only bias terms, we untie all the bias terms for each layer and expect to use the free bias terms to specialize the layers with tied weights. As BitFit, bias-based layer adaptation introduces very few additional parameters without inference overhead.
Adapter-based Layer Adaptation (Adapter-LA) Adapter-based approaches (Houlsby et al., 2019) introduce adapter modules for NLP task transfer, which achieve competitive results without full fune-tuning. We borrow this idea for layer adaptation by introducing an independent adapter module for each layer. Specifically, we adopt the recently proposed LoRA (Hu et al., 2021) that shows promising results for task adaptation in GPT-3 (Brown et al., 2020) as our layer adapter, as Figure 4(a) shows: where i denotes the ith layer, and W LoRA is a low-rank decomposition by A and B, as shown in Figure 4(a). In our experiments, we apply the layer adapter to W Q and W V , as the original paper of LoRA suggests.
Prompt-based Layer Adaptation (Prompt-LA) As recent work (Brown et al., 2020;Li & Liang, 2021;Lester et al., 2021) uses prefix/prompt for task transfer, we investigate if such an approach can work for layer adaptation. As Figure 4(b) shows, we introduce L tokens with learnable parameters as a specific prompt for each layer to adapt layers with tied weights. The prompts are only used for keys and values in attention modules, which will not introduce much inference overhead as long as L is not set to be too large.
Following the encoder-favored principle in Section 3.2, we only apply LA to encoder layers.

Experimental Setting
We evaluate in the two most practical on-device seq2seq tasks: Machine Translation (MT) and Grammatical Error Correction (GEC). In MT, we select the most popular MT benchmark -WMT14 English-German (En-De) translation task, which is also a touchstone for seq2seq evaluation, as our main test bed. To compare with previous work, we also evaluate WMT14 English-French (En-Fr) translation task. We follow the standard way to train and evaluate evaluate WMT14 En-De and En-Fr. As Ott et al. (2018), we use a joint source-target dictionary of 32K Byte Pair Encoding (BPE) for En-De, and 40K BPE for En-Fr. We mainly use sacreBLEU (Post, 2018) for evaluation.
In GEC evaluation, we follow the BEA-19 restricted setting to use Lang-8 (Mizumoto et al., 2011)  We select UNIVERSAL TRANSFORMER which is the most popular parameter-efficient Transformer as our main baseline for fair comparison. By default, we apply Seq-KD (Kim & Rush, 2016) to train models, since there is no reason for training smaller models in practice without Seq-KD that is effective to improve results with no harm. The full-parameterized 6+6 Transformer-big (d = 1, 024) model (Vaswani et al., 2017;Ott et al., 2018) is used as the teacher 5 . For inference, we use beam search (beam=5) by default.

Machine Translation
We evaluate EDGEFORMER and compare it with UNIVERSAL TRANSFORMER (UT) on WMT14 En-De. According to Table 2, the EDGEFORMER without layer adaptation (LA) largely outperforms UTs. Among the LA approaches, both Adapter-LA and Prompt-LA are clear to benefit the result with marginal computational or parameterization cost, while Bias-LA does not show significant performance gain though it is the cheapest.
As discussed in Section 3.2, the advantage of EDGEFORMER over UT is owing to its cost-effective parameterization. The encoder-favored principle is again supported by comparing the 6+6 Transformers' results in Table 2, which is consistent as the observation on the dev set in Figure 3a.
To further understand the effectiveness of load-balanced parameterization principles, we conduct an ablation study by adjusting encoder FFNs in EDGEFORMER. Table 3 shows the results of EDGEFORMER with various FFN parameterization. As we reduce d ffn (e.g., to 1536 or 1024), we can increase the group of encoder FFN parameters and reduce their load given a fixed parameterization budget. However, such a strategy leads to a clear degradation of sacreBLEU. One reason is that the FFN parameters of a reduced load (3-4 times) are not so fully utilized as the baseline (6 times) despite other reasons such as the differences of network shape (e.g., d ffn ). To minimize the effects of other factors, we compare the first group with a balanced parameter load (i.e., 6-6) and the last group with  Figure 5: The effects of (a) rank r in Adapter-LA, and (b) prompt length L in Prompt-LA on the performance in WMT14 En-De. Note that r = 64 will lead to exceed our parameterization budget although its performance is better.  a imbalanced parameter load (1-11 or 11-1), showing load-balanced parameterization is consistently better than the imbalanced counterparts.
After discussing parameterization, we then analyze the effects of layer adaptation on the results by mainly focusing on Adapter-LA and Prompt-LA that both show performance gain. Figure 5 shows the effects of the rank r in Adapter-LA and prompt length L in Prompt-LA. As r increases, the model performance will gradually improve. However, when r becomes large (e.g., r ≥ 64), it will exceed our parameterization budget and thus the gain will become meaningless. As for prompt length L in Prompt-LA, it is different from r that it will not keep improving the results as it increases: the gain can hardly be observed after some length (e.g., L = 8), which is similar to the observation in prefix-tuning (Li & Liang, 2021). Therefore, we use r = 32 and L = 8 as the default setting to report the results of Adapter-LA and Prompt-LA.
Finally, we compare EDGEFORMER with recent work on parameter-efficient Transformer modeling.
To keep consistency of the training and evaluation protocols with previous work, we here give up using Seq-KD to train the models, and report BLEU (Papineni et al., 2002) for comparison. Specifically, we compare with DeLighT (Mehta et al., 2020), Shapeshifter (Panahi et al., 2021) and Lite Transformer (Wu et al., 2020), and show the results in Table 4. However, it is notable that the results are not strictly comparable because the previous studies have their own focus and setting, which are different from ours. For example, DeLighT and Lite Transformer focus much more on FLOPS than the model size, thus they both exceed 6 the parameterization budget although they are fast; while Shapeshifter's goal is minimizing the model size despite an additional 10% ∼ 20% inference overhead. Regardless of these factors that prevent fair comparison, EDGEFORMER achieves very competitive results in En-De under the strict on-device resource constraints, which outperforms the  state-of-the-art Shapeshifter with the similar model size. It is notable that EDGEFORMERs do not perform so well in En-Fr as En-De, compared with Shapeshifter. One reason is that we use the same model architecture and training configuration for both En-De and En-Fr, while Shapeshifter uses different model architecture configurations specific for En-De and En-Fr.

Grammatical Error Correction
We show GEC results of EDGEFORMER in Table 5, and observe similar phenomena that EDGE-FORMER largely outperforms UT and layer adaptation further improves results. Among them, EDGEFORMER with Adapter-LA even outperforms the full-parameterized Transformer because a small training set (900K sentence pairs) is too hard to to truly show the capability of densely parameterized models, which is also pointed out by previous work (Mehta et al., 2020).

Runtime Evaluation
We conduct experiments for runtime latency and memory evaluation using onnxruntime 7 that supports efficient seq2seq decoding. We apply int8-quantization to EDGEFORMER and test latency on two devices: a 2-core 8 Intel® Xeon® E-2288G CPU with 4GB RAM, and a 2-core Qualcomm SM8150 Snapdragon 855 CPU in Pixel 4, which are both current mid-to-high end CPUs launched 2-3 years ago.  Table 6: Runtime results for int8-quantized EDGEFORMER, in which Latency1 and Latency 2 denote the average latency per sentence measured on the Intel® Xeon® E-2288G CPU and Qualcomm SM8150 Snapdragon 855 CPU, respectively. Note that in the runtime evaluation, we run through the test set with batch size=1, and use greedy decoding instead of beam search.

Pretrained EdgeFormer
Given the promising performance of EDGEFORMER, we preliminarily pretrain 9 EDGEFORMER (Adapter-LA) with 2K sentenpiece vocabulary through the same self-supervised task (i.e., masked span infilling) as T5 (Raffel et al., 2019) and make it publicly available for downstream English seq2seq tasks under the on-device resource constraints.
• Pretrained 12+2 UT (wide): the pretrained 12+2 Universal Transformer with larger FFNs (d ffn = 3, 072) which allows it to have the comparable number of parameters as EDGE-FORMER but leads to more FLOPS.  According to Table 9, the pretrained EDGEFORMER achieves significantly better performance than the pretrained UT models as well as the Transformer-base model trained from scratch. We believe that the pretrained EDGEFORMER, as the first publicly released on-device seq2seq pretrained model, can largely facilitate on-device seq2seq generation in practice.

Related Work
On-device seq2seq generation is a research area that has been much less explored than on-device CV. Besides the general techniques like quantization, pruning and knowledge distillation, there are some research branches closely related to this paper that focus on seq2seq learning. Among them is parameter-efficient Transformer modeling. In this branch, UNIVERSAL TRANSFORMER (Dehghani et al., 2018) uses cross-layer sharing method, which is the most popular way for saving parameters, although its original motivation is not saving parameters. Takase & Kiyono (2021) extends UNIVER-SAL TRANSFORMER by studying different ways for layer sharing, and Reid et al. (2021) proposes to free the first and last encoder layer and widen the intermediate layers for better performance. However, both the approaches consider parameter-efficiency only without caring decoding latency that becomes much worse.
In addition to work improving parameter efficiency by weight sharing, there is research that studies lightweight model architecture for seq2seq learning where early work (Gehring et al., 2017;Wu et al., 2019) mainly focuses on CNNs, while recent efforts have tended to switch to attention-based models such as Mehta et al. (2020). Also, low-rank factorization has been studied intensively to make the model tiny (Zhang et al., 2021;Panahi et al., 2021); and hardware-aware network architecture search with elastic modeling (Wang et al., 2020) has been proposed recently for facilitating deployment of seq2seq models on various devices. Among previous studies, the work of Wu et al. (2020) is the most related to ours, which studies seq2seq generation in a mobile setting. However, it sets the computational constraint for on-device seq2seq to be the same with the CV tasks, which is too strict and unnecessary, as discussed in Section 2.2. As a result, either models that they studied are too large and cost massive memory or fail to achieve desirable performance despite high inference speed. In contrast, our work carefully evaluates the bottleneck constraints, and proposes appropriate models with parameterization and layer adaptation innovations, largely improving the results for practical on-device seq2seq generation.

Conclusion and Future Work
In this paper, we formally study on-device seq2seq generation problem, including defining its resource constraints and proposing an appropriate model EDGEFORMER. The cost-effective parameterization and layer adaptation innovations in EDGEFORMER prove effective to improve the results with negligible computation and memory cost, achieving state-of-the-art results in the on-device seq2seq generation setting. Our released pretrained EDGEFORMER checkpoint can be easily fine-tuned for English seq2seq tasks, largely facilitating on-device seq2seq generation in practice.
For future work, we plan to further study load-balanced parameterization for parameter-efficient models, which is an interesting and new but seemingly profound machine learning research problem: instead of naively assuming that all the parameters are equal in this preliminary study, we suspect that parameters in different modules (e.g., parameters in the self-attn and FFN; or parameters in different layers) should be under different amounts of load. We look forward to in-depth research on this problem, which might be helpful to deepen our understanding of neural networks.  A Details of datasets and training configuration for machine translation and grammatical error correction For MT, we follow the setting of Ott et al. (2018) to train the model on the WMT14 datasets. For En-De, the training set contains 4.5M parallel sentence pairs. We use newstest2013 as our dev set. For En-Fr, there are 36M parallel sentence pairs for training, and we use newstest2012+2013 as the dev set.
For model training configuration, we show in Table 8:

B Details of pretraining and fine-tuning
We pretrain EDGEFORMER with the same pretrain data as RoBERTa  through the same pretrain task as T5 (Raffel et al., 2019). The details of pretraining are shown in Table 9.
For downstream task fine-tuning, we present the details at our model/code released page.