{'fraction_non_alphanumeric': 0.03264626808347203, 'fraction_numerical': 0.012642427346050441, 'mean_word_length': 4.793621361023549, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Grammatical Error Correction Systems for Automated Assessment: Are They Susceptible to Universal Adversarial Attacks?

Grammatical error correction (GEC) systems are a useful tool for assessing a learner’s writing ability. These systems allow the grammatical proficiency of a candidate’s text to be assessed without requiring an examiner or teacher to read the text. A simple summary of a candidate’s ability can be measured by the total number of edits between the input text and the GEC system output: the fewer the edits the better the candidate. With advances in deep learning, GEC systems have become increasingly powerful and accurate. However, deep learning systems are susceptible to adversarial attacks, in which a small change at the input can cause large, undesired changes at the output. In the context of GEC for automated assessment, the aim of an attack can be to deceive the system into not correcting (concealing) grammatical errors to create the perception of higher language ability. An interesting aspect of adversarial attacks in this scenario is that the attack needs to be simple as it must be applied by, for example, a learner of English. The form of realistic attack examined in this work is appending the same phrase to each input sentence: a concatenative universal attack. The candidate only needs to learn a single attack phrase. State-of-the-art GEC systems are found to be susceptible to this form of simple attack, which transfers to different test sets as well as system architectures,


Introduction
Grammatical Error Correction (GEC) systems can form a part of automated language fluency assessment: the number of edits from a candidate's input sentence to a GEC system's grammatically corrected output sentence is indicative of a candidate's language ability, where fewer edits suggest better fluency. Early GEC systems were designed using hand-crafted rules (Naber, 2003), but since, data driven approaches, such as Statistical Machine Translation (Yuan and Felice, 2013), emerged. With encoder-decoder architectures dominating in Neural Machine Translation, Yuan and Briscoe (2016) used Recurrent Neural Networks (Cho et al., 2014) to improve GEC performance. Now state of the art GEC systems are based on the Transformer (Vaswani et al., 2017) architecture (Kaneko et al., 2020;Chen et al., 2020;Malmi et al., 2019;Awasthi et al., 2019;Omelianchuk et al., 2020b;Kiyono et al., 2019;Lichtarge et al., 2020;Stahlberg and Kumar, 2020).
Despite the success of Transformer-based deep learning systems, there is a shortcoming: Szegedy et al. (2014) discovered that neural networks are susceptible to adversarial attacks, where a small change at the input can yield large, undesired changes at the output of the model. In the GEC setting, a candidate may seek to make a change to their input sentence, such that the system makes no corrections, resulting in zero edits between the source and prediction sequences, which falsely indicates perfect language fluency. Given the highstakes of an assessment setting, it is particularly concerning if a candidate can engage in such malpractice. Hence, this work explores the susceptibility of GEC systems to adversarial attacks.
GEC systems operate on natural language inputs. In this domain, there are many proposed adversarial attacks (Zhang et al., 2019), but on the whole they are inappropriate for sequenceto-sequence tasks, such as GEC. Ebrahimi et al. (2018); Zou et al. (2019); Zhang et al. (2021); Cheng et al. (2018) introduced methods for adversarial attacks in sequence-to-sequence models. These works require multiple queries of the target system. However, a candidate cannot query a GEC system. To solve this issue, this work uses a universal (Moosavi-Dezfooli et al., 2016) adversarial attack. Here, the same universal attack phrase is appended to the end of all candidates' input sen-tences, i.e. a new candidate can simply acquire (e.g. through purchase) a fixed universal attack phrase to concatenate to their input and deceive a GEC system used for automatic fluency assessment. This work also considers the transferability of a single attack phrase across different datasets and even architectures. Further analysis is carried out to determine the aspects of GEC systems that cause them to be susceptible to this form of attack.
Despite advances in natural language adversarial attacks, there has been less research on developing defence schemes. Defence strategies can be categorized as model modification, where the model or data is altered at training time (e.g. adversarial training (Yoo and Qi, 2021)) or detection (Raina and Gales, 2022), where external systems or algorithms are applied to trained models to identify adversarial attacks. Model modification approaches demand re-training of models and so detection approaches are preferred for deployed systems. Note that for attacks on GEC systems, detectors based on grammatical (Sakaguchi et al., 2017) and spelling (Mays et al., 1991;Islam and Inkpen, 2009) errors will fail. In this work, the most popular detection approaches: Frequency Guided Word Substitution (Mozes et al., 2020) (shown to outperform ); perplexity (Han et al., 2020;Minervini and Riedel, 2018) and model confidence (Aldahdooh et al., 2021); are applied to detecting adversarial attacks on GEC systems.

Related Work
In literature there has been limited work examining adversarial attacks for GEC systems. However, some works have explored adversarial robustness. First, Wang and Zheng (2020) perform adversarial training to improve the performance of their GEC system. Their adversarial training scheme augments the training data with adversarial examples, generated through the insertion of common grammatical mistakes in grammatically correct sentences, where the insertions are tuned to exploit weak spots in the GEC system. Further, Tang (2021) also seeks to increase robustness of GEC systems in a post-training setting, through further training on adversarial examples generated from four different NLP adversarial attack schemes. These adversarial attack methods again are designed to fool the sequence-to-sequence GEC system. Finally, Farkas et al. (2021) also augment the  training data with adversarial examples, but focus   on ensuring the adversarial examples mimic human  grammatical errors by introducing noise at both a  token level and embedding level. However, the above schemes are inappropriate for the attack setting in this work. First, the aim of the attack in this work is to perturb grammatically incorrect sentences to conceal grammatical errors. Second, the existing works consider attacks specific to each input, whereas this work considers the more realistic setup of a universal adversarial attack.

Grammatical Error Correction
Grammatical Error Correction (GEC) systems perform a sequence-to-sequence task, where an input word sequence, x 1:T , containing grammatical errors, is corrected for these errors by the system, with parameters, θ to predict the grammatically correct output word sequence,ŷ 1:L , y 1:L = arg max y 1:L {p(y 1:L |x 1:T ; θ)}. (1) To evaluate the performance of a GEC system, it is necessary to identify the edits made by the system and compare to the reference edits. An edit is defined as a modification (insertion, deletion or substitution) required on the input sequence x 1:T to make it match the target sequence, y 1:L . A popular edit extraction tool is ERRANT (Bryant et al., 2017), which uses a linguistically-enhanced alignment algorithm proposed by Felice and Briscoe (2015). Edits between the input sequence, x 1:T , and hypothesised prediction sequenceŷ 1:L can be found,ê 1:P ,ê 1:P = edits(x 1:T ,ŷ 1:L ). ( These edits are to be compared to reference edits, whereỹ 1:L is the reference output sequence. The precision = TP/(TP+FP) and recall = TP/(TP+ FN) can now be computed, where TP, FP and FN are the standard definitions of true-positive, falsepositive and false-negative. As a single performance score, F 0.5 = 1.25 * prec * rec/(0.25 * prec)+ rec) is used, giving greater weight to precision over recall, as in GEC systems it is more important to be correct in the hypothesised edits,ê 1:P , as opposed to identifying all reference edits,ẽ 1:R . In this work GEC systems are considered for automated assessment. Here, the fluency score, S θ (x 1:T ), of a candidate is measured by the count of edits between the input sequence, x 1:T , and hypothesised prediction sequenceŷ 1:L , i.e. S θ (x 1:T ) = count(ê 1:P ) = P, where S θ (x 1:T ) = 0 is a perfect fluency score. Beyond extracting edits and reporting the overall performance of a GEC system, it is useful to categorize the error types. Inspired by Swanson and Yamangil (2012), the ERRANT tool uses a rulebased error type framework. Here edits are classified as either: Missing, where a token is present in the target sequence, y 1:L but not in the input sequence, x 1:T ; Replaced, where a substitution is made; or Unnecessary, representing edits where a token is present in the input sequence, x 1:T and not the output target sequence, y 1:L .

GEC Adversarial Attack
A targeted adversarial attack on an input text sequence, x 1:T aims to perturb it to generate an adversarial example x ′ 1:T ′ that ensures the output of a classifier, F(), is t, H() is some distance metric between the original and adversarial input sequences, ensuring the change is imperceptible. It is not simple to define an appropriate function H() for word sequences. Perturbations can be measured at a character or word level. Alternatively, the perturbation could be measured in the vector embedding space, using for example l p -norm based (Goodfellow et al., 2015) metrics or cosine similarity (Carrara et al., 2019). However, constraints in the embedding space do not necessarily achieve imperceptibility in the original word sequence space. This work uses a simple variant of a Levenshtein edit-based measurement (Li et al., 2018) which counts the number of changes between the original sequence, x 1:T and the adversarial sequence x ′ 1:T ′ , where a change is a swap/addition/deletion, and ensures it is smaller than a maximum number of changes, N . For a candidate planning to perturb their input sentence, the simplest attack is concatenation, where a fixed phrase is appended to their input (Wang and Bansal, 2018;Blohm et al., 2018;Raina et al., 2020), The aim of the adversarial attack on a GEC system used for automated assessment, F() = S θ () (Equation 4), is to maximally decrease the count of edits between the input sequence and the predicted sequence, i.e. a candidate wants to conceal their grammatical errors from the GEC system. A single universal adversarial phrase,δ 1:N is to be used for all candidates, i.e. once this universal phrase has been learnt from a set of J candidates, it can be sold to other candidates. Hence, the cost function an adversary seeks to optimise iŝ δ 1:N = arg min where V k is the set of all k length word sequences that can be constructed from a selected language vocabulary, V.
It is important to consider the interpretation of imperceptibility in the automated assessment setting. In many applications, measuring imperceptibility by counting number of added words, N , is inadequate as it can result in incomprehensible phrases that can easily be identified by a human reader. However, in this setting, there is no human reader, which demands the use of automated systems for identifying incomprehensible phrases. Therefore, this work includes experiments to filter for adversarial attack words that do not compromise the integrity of an input sentence, when measured using a perplexity detector (introduced as a detection mechanism in Section 5, Equation 9) based on a state of the art language model. This ensures that an attack phrase remains imperceptible in an automated assessment setting.
This work also investigates variations in the punctuation a candidate can use to concatenate an adversarial phrase to an input sentence. If '*' represents the form of punctuation, then to concatenate an adversarial phrase to the original phrase, we do: original phrase* adversarial phrase.

Defence
For deployed systems, defence strategies that require re-training are undesirable. It is easier to use detection processes to identify and flag adversarial examples. This section considers how state of the art detection approaches can be applied to universal concatenation adversarial attacks on GEC assessment systems, described in Section 4.
All detection approaches, D(), use a selected threshold, β to classify an input sequence, x 1:T as adversarial or not. When D(x 1:T ) > β, then the input sequence x 1:T is flagged as an adversarial example. To examine the performance of the detection process, this work uses precision-recall curves, where precision and recall values are calculated for a sweep over the threshold β. Here, for each value of β, the precision and recall values are calculated (as in Section 3), with adapted definitions for true-positive (number of samples correctly classified as adversarial), false-positive (number of samples incorrectly classified as adversarial) and false-negative (number of samples incorrectly classified as non-adversarial). A single-value summary is again obtained with the F 0.5 score, giving greater weighting to precision over recall, as it is more important to be correct in accusing candidates of mal-practice than finding all the candidates that cheat. The threshold with the highest F 0.5 score is selected for the detector D().
The recently dominating, Frequency Guided Word Substitution (FGWS) (Mozes et al., 2020) algorithm is adapted for attacks on an assessment GEC system. For the FGWS algorithm, we generate a sequence x * 1:T from the original input sequence, x 1:T by substituting out low frequency words for higher frequency words. Precisely, a subset of eligible words (for substitution) is found gives frequency of word x and γ ∈ R >0 is a frequency threshold. Then, for each eligible word x ∈ X E a set of replacement candidates, U(x) is found using synonyms. A replacement word x * is selected as x * = arg max w∈U (x) ϕ(w). Hence, x * 1:T is generated by replacing each word x in x 1:T if ϕ(x * ) > ϕ(x). For the GEC assessment system, S θ (), defined in Equation 4, the FGWS detection score is, Smith and Gal (2018) describe the use of uncertainty for adversarial attack detection, where adversarial samples are thought to result in greater epistemic uncertainty. In this work, negative confidence is selected as a simple measure of uncertainty. It is easiest to measure the confidence using the grammatically correct sequence output by the GEC system,ŷ 1:L (Equation 1). The negative confidence detector score is calculated as, This works also explores the positive confidence detector, D pc (x 1:T ) = −D nc (x 1:T ). A final popular NLP detection approach is to consider the perplexity (Minervini and Riedel, 2018) of the input sequence. It is expected that adversarial sequences have a greater perplexity than original samples. The perplexity detector, using some language model (LM), can be defined as, In recent years, Grammatical Error Correction systems have been dominated by large (up to 11B parameters) Transformer based architectures (Rothe et al., 2021;Stahlberg and Kumar, 2021). Using the F 0.5 metric defined in Section 3, Table 1 compares the performance of two popular Transformer-based architectures: the Gram-former (Damodaran, 2022) (223M parameters), a T5-based (Raffel et al., 2019) sequence to sequence system 3 and Grammarly's Gector (Omelianchuk et al., 2020a), using specifically the Roberta-based architecture ) (123M parameters) 4 . The Gramformer is pre-trained on the WikEd Error Corpus (Grundkiewicz and Junczys-Dowmunt, 2014), and in this work, it is further fine-tuned on the CLC (with FCE-test set removed), WI and LOCNESS datasets. The finetuning uses Adam optimiser with a batch size of 256 and a learning rate of 5e-4 with warm up. Maximum sentence length is set at 64 and the final model parameters are averaged over 5 best checkpoints. As the Gramformer model was initialised from a large pre-trained system, changing seed for the finetuning gave little diversity in the ensemble. Table 1 shows that the Gramformer and Gector systems have a similar performance on the FCE test set, but the Gector system significantly out performs the Gramformer on the CoNLL-14 and BEA-19 test sets. Nevertheless, to mimic a realistic adversarial attack setting, the more easily available Gramformer system 5 is used as an initial model (adversary can access) for learning universal attacks and the best attacks are then transferred for evaluation on the target Gector system in Section 6.4.

Attack Results
Greedy universal concatenation adversarial attacks were performed on the Gramformer system as described in Equation 6. As described in Section 4, different punctuation types were considered for the concatenation of the universal attacks. The impact of each attack phrase is presented for each of the three different GEC test sets in Figure 1,  being the number of universal adversarial words at the end of each input sentence. The universal attack phrases were learnt on the FCE training split 6 . The metric used to measure the success of the attack is the fraction of samples with zero edits from source to GEC prediction sequence. The random attacks shown use a full-stop for concatenating randomly sampled words. A direct attack is where no punctuation is used to separate the original and the attack phrase. With percent increases between 20% and 50% in the fraction of samples with no edits shows that the GEC system is threatened somewhat by the direct, colon and comma attacks. However, for the full-stop universal adversarial attack sequence, with even a N = 4 word attack, the number of samples with zero edits increases by almost 40% for the FCE test set and more than 100% for the CoNLL-14 and BEA test set. It is evident that the GEC system is susceptible to even a simple form of universal attack. The greater susceptibility to the full-stop attack can be explained to some extent by the nature of the data used to fine-tune the Gramformer GEC system. Table 2 shows the frequency count of the different punctuation marks in the training set (CLC, WI and LOCNESS datasets), where the full-stops present at the end of sentences are not included 7 . Note that there are a total of ∼3M input samples in the training dataset. The count of full-stops is far less than that of commas, meaning the GEC system is not as familiar with multi-sentence inputs allowing for greater susceptibility to attacks using the full-stop. However, this count-based explanation is inadequate to justify the less successful colon concatenation attack. Nevertheless, the lack of susceptibility to colon concatenation can be explained -in the training samples with colons, more than 50% samples have the colon followed by a list delimited with semi-colons. This means that the GEC system easily learns this fixed colon usage, which makes it difficult to have a successful colon-based universal concatenation attack format. Due to the potency of the full-stop concatenation attack, the remainder of the analysis in this section focuses on the full-stop attack 8 . Examples of the impact of the universal attack are given in   Table 3 shows the impact of the N = 4 concatenation adversarial attack on the performance of the GEC system on the FCE test set. The adversarial phrase is stripped from the output predicted sequence to discount the introduction of false-positive edits in the adversarial part of the input. As expected the F 0.5 score worsens due to the drop in the recall, i.e. the GEC systems struggles to find the grammatical errors with the attack phrase concatenated at the end of the sentence -the attack is successful in concealing the errors present in the sentence.

Detection Evasion
Although the Gramformer GEC system is susceptible to a universal attack, it can be defended using detection methods. Figure 2 compares the success of detectors from Section 5 when attempting to distinguish adversarial samples from original samples (on FCE test). The threshold for each detector is selected such that it gives the best F 0.5 score. Results are presented for original FCE test samples with and without the full-stop universal adversarial phrase appended to the end of the samples. It is interesting to note that FGWS, although successful in other NLP adversarial attack tasks, has little success here. This is perhaps expected as the FGWS vocabulary is now trained with grammatically incorrect sentences containing mis-spellings. Further, the FGWS algorithm is tuned to word substitution attacks, meaning it is less appropriate for the concatenation setting here. The perplexity score is calculated using a pre-trained distilled GPT-2 language model (Radford et al., 2019) applied to the input sequence. Perplexity has some success here in detecting adversarial samples, but the success is limited because many original input sequences are grammatically incorrect and thus naturally have an inflated perplexity score, meaning it is easy for the detector to mistake them for adversarial samples.
Interestingly, negative confidence has no success in detection here, whilst positive confidence dominates as the best detection approach. This is surprising because one would expect adversarial samples to cause systems to be less confident in their predictions, as the system is operating in a less well understood input space. Nevertheless, superior performance of positive confidence is explainable. GEC systems are trained on data where the tokens present in the input are also present in the reference, meaning in most cases there is a strong bias towards simply predicting tokens that are present in the input sequence. When an obscure adversarial word is present in the input sequence, the GEC system at prediction time naturally has a much larger probability mass associated with this obscure word, i.e. it is excessively confident in predicting it. An adversary may have knowledge of the powerful detectors used here and would tailor the adversarial attack to avoid detection. Figure 3 shows the impact of the greedy attack approach modified to evade detection from the confidence detector and the perplexity detector (detector thresholds set to the value corresponding to the F 0.5 score in Figure  2) 9 . The attack phrases are learnt on the FCE train set and evaluated on the FCE, CoNLL-14 and BEA test sets. It is interesting to note that the confidence detection evading attack phrases are only slightly less effective than the original attack phrases -the fraction of zero edits saturate at around 0.50 as opposed to 0.56 (on FCE test set). However, considering the attack to evade the perplexity detector, the potency of this universal phrase is surprisingly greater than the original greedy attack phrase learnt (for all datasets). This suggests that constraining an attack to more human phrases (as measured by perplexity of a powerful GPT-2 language model), allows for even stronger adversarial attacks. These phrases are considered particularly threatening as their similarity to natural language allows for greater imperceptibility to human observers (not just automated detection systems).

Transfer Attack
The aim of this section is to investigate the impact of transferring an attack learnt for an initial system (Gramformer) to a target system (Gector).
Concatenation universal adversarial attacks on the Gramformer system are found to be most powerful when the adversary greedily generates a phrase 9 A adversarial word is accepted if the average confidence/perplexity is less than the detector threshold. that evades a perplexity detector, as demonstrated in Figure 3. Hence, this universal adversarial phrase is simply evaluated on the Gector system. The results in Table 4 show that this transferred universal adversarial phrase has some level of threat: across all test sets, this universal adversarial phrase is able to increase the fraction of samples with no edits by at the least 10%. Table 4 also gives the impact of learning a universal attack phrase (using FCE train dataset and also avoiding a perplexity detector as in Section 5) for the Gector system. Interestingly, the direct attack is only around twice as effective as the transferred attack. This highlights the potency of these forms of adversarial attacks: the same adversarial phrase can transfer to different unseen, GEC systems.

Analysis
This section carries out a more in-depth analysis to understand the aspects of the GEC systems exploited by adversarial attacks. The analysis presented here is for the concatenative full-stop attack learnt for the Gramformer system. We want to analyse the nature of the attackprecisely which type of edits is the adversarial attack phrase targeting. If for a dataset of J inputreference sentence pairs, there exist a total R reference edits,ẽ 1:R (Equation 3) and P hypothesis edits,ê 1:P (Equation 2), then the performance due to the GEC system correctly hypothesising edits can be measured by the correction rate, cor and the failure measured by the insertion rate, ins, where {ẽ 1:R } ∁ gives the complement set. Section 3 classifies an edit as Missing, Replaced or Unnecessary. Figure 4 shows how the correction and insertion rates change (on FCE test) for each of these edits classes separately. Note that there are a total of R = 919, R = 2954 and R = 596 reference edits for Missing, Replaced and Unnecessary classes respectively. The edit classes (M, R and U) all undergo a similar drop in correction rate with an increasingly powerful adversarial attack. However, Figure 4b demonstrates that smaller N adversarial attacks struggle to reduce Unnecessary inserted edits more than other edit type classes. Only when the reductions from removing Missing and Replaced inserted edit types have saturated, does increasing N reduce the Unnecessary inserted edit types. The flattening of the performance curve (fraction of samples with zero edits) suggests that this reduction in inserted Unnecessary edits has little benefit to the adversarial attack. The apparent robustness of Unnecessary inserted edits can perhaps be explained simply. An inserted edit is the creation of an edit,ê, by the GEC system that is not present in the reference edits,ẽ 1:R . When a GEC system creates specifically Unnecessary edits it means a token present in the input sequence is not present in the output prediction sequence. A well trained GEC system will remove the adversarial phrase appended to the input sequence, creating an Unnecessary edit,ê, which does not exist in the reference edits,ẽ 1:R -it is an inserted edit. Hence, there is an artificial increase in inserted Unnecessary edits. Edits in the adversarial phrase only contribute to 4% of total edits on average (analysis presented in Figure  A.1), where 91% of the adversarial phrase edits are Unnecessary edit types. This gives on average an increase in the inserted Unnecessary edit rate by 10% (0.04 * 0.91 * count(ê 1:P )/596), where 596 is the count of Unnecessary reference edits. This increase of 10% explains the shift between the Replaced and Unnecessary curves in Figure  4b. Hence, all edit types in an input sequence are susceptible to the simple universal attack.

Conclusions
Grammatical Error Correction (GEC) systems can contribute to automated fluency assessment. The count of edits between a candidate's input and the grammatically correct output sequence from the GEC system, is a measure of the candidate's ability in the language: fewer the number of edits, the better the candidate. However, this work showed that deep learning based GEC systems are susceptible to adversarial attacks, where a candidate can cheat by adjusting their input sentence such that the predicted sequence from the GEC system does not correct the existing grammatical errors.
To model a realistic adversarial attack setting, this work restricts itself to a blackbox, universal attack approach, where the same adversarial phrase is appended to the end of all candidates' input sequences. This setting is particularly threatening because a candidate can cheat without querying the GEC system even once -the candidate only has to acquire the attack phrase. It is found that the same universal attack phrase can be effective across multiple datasets and more interestingly can be transferred to deceive new, unseen architectures. This demonstrates that all GEC systems have a worrying susceptibility to even the simplest attack forms.

Limitations
This work identified methods to adversarially attack state of the art GEC systems. Defence strategies in the form of detection were also considered. However, there has been less focus on adversarial training to improve robustness of systems. Although adversarial training is not an option available to deployed GEC systems, future work in this area would be useful in understanding the increase in robustness from adversarial training to the universal attack form considered in this work.

Risks and Ethics
Adversarial attacks, by nature, are of ethical concern in high stakes' environments. The approaches proposed in this work can be used to inspire candidates to engage in mal-practice in an education setting. However, this is of little concern because the development of attacks requires significant knowhow of the GEC assessment process, which candidates are unlikely to have.

A More Analysis
It is perhaps surprising to observe a reduction in the number of edits made by a GEC system, when a potentially grammatically incorrect adversarial phrase is concatenated at the end of each input. To investigate this, Figure A.1 presents the split in the number of edits; i.e. on average how many edits are found in the original part of the input and the concatenated adversarial part of an input separately (using the full-stop concatenation attack on the Gramformer evaluated on the FCE test set).
The adversarial phrase appears to have a length independent small contribution to the total number of edits, whilst the larger reduction in the number of original part edits allows for an overall decrease in the total edits.

Input
I have received your letter, which was about that you would like me to inform you. ink. Pred I have received your letter, which was about what you would like me to inform you about. Adv Pred I have received your letter, which was about what you would like me to inform you. ink.

Input
When I was a child I always wanted to get the lyrics of the songs that I liked, but it was nearly impossible to have them. ink. Pred When I was a child I always wanted to get the lyrics of the songs that I liked, but it was nearly impossible to get them. Adv Pred When I was a child I always wanted to get the lyrics of the songs that I liked, but it was nearly impossible to have them. ink.

Input
As you know, it is not suitable to wear a jean or something. ink.

Pred
As you know, it is not fashionable to wear jeans or something. Adv Pred As you know, it is not suitable to wear jeans or something. ink.