{'fraction_non_alphanumeric': 0.04304015767430402, 'fraction_numerical': 0.01729490022172949, 'mean_word_length': 4.916192974785017, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales

Neural language models (LMs) have achieved impressive results on various language-based reasoning tasks by utilizing latent knowledge encoded in their own pretrained parameters. To make this reasoning process more explicit, recent works retrieve a rationalizing LM's internal knowledge by training or prompting it to generate free-text rationales, which can be used to guide task predictions made by either the same LM or a separate reasoning LM. However, rationalizing LMs require expensive rationale annotation and/or computation, without any assurance that their generated rationales improve LM task performance or faithfully reflect LM decision-making. In this paper, we propose PINTO, an LM pipeline that rationalizes via prompt-based learning, and learns to faithfully reason over rationales via counterfactual regularization. First, PINTO maps out a suitable reasoning process for the task input by prompting a frozen rationalizing LM to generate a free-text rationale. Second, PINTO's reasoning LM is fine-tuned to solve the task using the generated rationale as context, while regularized to output less confident predictions when the rationale is perturbed. Across four datasets, we show that PINTO significantly improves the generalization ability of the reasoning LM, yielding higher performance on both in-distribution and out-of-distribution test sets. Also, we find that PINTO's rationales are more faithful to its task predictions than those generated by competitive baselines.


INTRODUCTION
Many language-based reasoning tasks require retrieving and reasoning over knowledge beyond the task input-e.g., commonsense reasoning and closed-book QA (Fig. 1, left) (Talmor et al., 2018;Mihaylov et al., 2018). Neural language models (LMs) have achieved impressive results on such tasks by utilizing latent knowledge encoded in their pretrained parameters (Raffel et al., 2020b;Brown et al., 2020). Still, given LMs' black-box nature, it is unclear whether this knowledge is being used properly (Doshi-Velez & Kim, 2017;Lipton, 2018). Previous studies have shown that LMs often learn spurious correlations from artifacts in downstream training data, thus limiting their generalizability (Branco et al., 2021;Geirhos et al., 2020;D'Amour et al., 2020).
With this in mind, a number of prior works aim to make LMs' reasoning processes more explicit by generating free-text rationales, which use LMs' internal knowledge to describe a reasoning process in natural language (Narang et al., 2020;Wei et al., 2022b;Marasović et al., 2022;Zelikman et al., 2022). In the fine-tuned self-rationalizing paradigm, a single LM is fine-tuned to jointly generate the task output and rationale (Narang et al., 2020;Marasović et al., 2022;Zelikman et al., 2022). In the prompted self-rationalizing paradigm, a single LM is instead frozen and prompted to jointly generate the task output and rationale, with the prompt consisting of a few input-output-rationale demonstrations (Wei et al., 2022b). In the pipeline-rationalizing paradigm, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to generate the output (Kumar & Talukdar, 2020;Rajani et al., 2019).

4) PINTO
StrategyQA:  However, when considering generalization performance, reliability, and deployment costs, these existing paradigms all have key limitations. Fine-tuned self-rationalizing LMs often perform worse than non-rationalizing LMs, since their parameters are learned using two relatively dissimilar objectives, while also requiring expensive rationale annotations (Wiegreffe et al., 2020;Narang et al., 2020). Prompted self-rationalizing LMs yield strong task performance and only need a few rationale demonstrations for the prompt, but are computationally prohibitive since they generally require very large-scale (i.e., over 100B parameters) LMs to work effectively (Wei et al., 2022a;b). Besides requiring expensive rationale annotations, pipeline-rationalizing LMs' generated rationale forms a non-differentiable bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020;Hase et al., 2020). Moreover, none of these paradigms has a mechanism for regularizing the rationale generation to faithfully reflect the reasoning process of the LM, without hurting task performance.
In this paper, we propose Prompted RatIonalizing with CouNTerfactual ReasOning ( PINTO), an LM pipeline that rationalizes via prompt-based learning, then reasons over the task input and rationale via counterfactual regularization. PINTO's rationalizing module is a medium-scale (i.e., 20B parameters) LM that contains vast latent knowledge obtained via pretraining (Black et al., 2022). Though prohibitive to fine-tune, it is affordable for prompt-based learning. Given the task input and a minimal input-output-rationale demonstration prompt, the rationalizing module uses its internal knowledge to map out a suitable reasoning process for the task input by generating a free-text rationale. The rationalizing module is frozen during fine-tuning, which drastically reduces training costs and prevents it from exploiting spurious shortcuts in the downstream training data. PINTO's reasoning module is a small-scale (i.e., under 1B parameters) LM to which knowledge is transferred from the rationalizing module. The reasoning module is fine-tuned to solve the downstream reasoning task by using the generated rationale as context for the task input. Crucially, to help ensure that the reasoning module's behavior is dictated by the rationale (instead of by spurious shortcuts), the reasoning module is regularized to output less confident predictions when the rationale is noisily perturbed. To simulate shortcut reasoning, we consider two rationale perturbation strategies: token masking (i.e., rationale is ignored) and token replacement (i.e., rationale is misused).
Across four question answering datasets (CSQA, StrategyQA, OpenBookQA, QASC), we show that PINTO significantly improves the reasoning LM's generalization, yielding higher performance on both in-distribution (ID) and out-of-distribution (OOD) test sets. Also, we find that rationales are utilized more faithfully by PINTO than by other methods, leading to better performance in lowresource settings. Furthermore, we show that PINTO's counterfactual regularization improves the reasoning module's robustness to noise in the rationalizing module's generated rationales.

RATIONALE-BASED LANGUAGE REASONING
In this work, we study LMs' ability to reason about language using implicit knowledge. We consider a specific type of multi-choice question answering (QA) tasks where the required knowledge for answering the question is not explicitly provided in the input and needs to be inferred from the LM's parameters (Talmor et al., 2019;Khot et al., 2020): Given a question q and a set of answer choices A = {a i }, the model's goal is to predict a plausibility score ρ(q, a i ) for each (q, a i ) pair, so that the predicted answerâ = arg max ai∈A ρ(q, a i ) matches the correct answer choice a * ∈ A.
Motivated by LMs' common tendency to exploit reasoning shortcuts when solving tasks (Branco et al., 2021), we focus on methods that explicitly generate free-text rationales to explain their predictions. Whereas extractive rationales are limited to input token scoring (Denil et al., 2014;Sundararajan et al., 2017;Chan et al., 2022b), free-text rationales use natural language to describe a reasoning process (e.g., with knowledge beyond the task input) (Narang et al., 2020;Wei et al., 2022b). Below, we discuss several paradigms (see also Fig. 1) for rationale-based language reasoning.
Fine-Tuned Self-Rationalization In this paradigm, an LM is fine-tuned to autogregressively generate the task output and rationale as a single sequence (Narang et al., 2020;Liu et al., 2018). If the rationale is generated after the task output, then the rationale is conditioned on the task output, and vice versa. Since the LM parameters are shared across two relatively dissimilar objectives, they often perform worse than non-rationalizing LMs (Wiegreffe et al., 2020;Narang et al., 2020). Notably, this paradigm requires expensive rationale annotations for all training instances.
Prompted Self-Rationalization In this paradigm, a pretrained LM is frozen and prompted to autogregressively generate the task output and rationale as a single sequence, with the prompt consisting of a few input-output-rationale demonstrations (Lampinen et al., 2022;Wei et al., 2022b). If the rationale is generated after the task output, then the rationale is conditioned on the task output, and vice versa. This paradigm performs well and only needs a few rationale annotations for the prompt, but it is computationally prohibitive since it generally requires very large-scale (i.e., over 100B parameters) LMs to work effectively (Lampinen et al., 2022;Wei et al., 2022b).
Pipeline Rationalization In this paradigm, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to predict the task output (Kumar & Talukdar, 2020;Rajani et al., 2019). Here, the generated rationale forms a discrete (i.e., non-differentiable) bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020;Hase et al., 2020). Additionally, the dedicated rationalizing LM requires extra rationale annotation/computation costs.

PINTO: FAITHFUL LANGUAGE REASONING USING PROMPTED-GENERATED RATIONALES
PINTO is a two-stage, rationalize-then-reason pipeline, designed to address the limitations of existing paradigms for rationale-based language reasoning ( §2). Like the pipeline rationalization paradigm, PINTO has separate modules for rationalizing and reasoning. However, PINTO's rationalizing module is prompted instead of fine-tuned. As a result, PINTO does not suffer from the non-differentiable bottleneck issue and has lower rationale annotation/computation costs.
Following prior works, PINTO is based on choice-specific rationales (Kumar & Talukdar, 2020;Hase et al., 2020). First, given q and A, the rationalizing module generates a set of choice-specific rationales R = {r i }, where each r i explains a reasoning process that supports answer choice a i ∈ A ( §3.1), as opposed to generating one rationale per question. We opt for this design choice because rationales are often answer-leaking (Sun et al., 2022), i.e., the rationale itself is already sufficiently predictive of one of the answer choices. If the rationalizing module only generates one rationale per question, then it is forced to make an "early decision" on the predicted answer, such that the reasoning module would only be left to recover the answer from the rationale (Kumar & Talukdar, 2020). While prior works require expensive rationale annotations to train/prompt the rationalizing module (Kumar & Talukdar, 2020;Hase et al., 2020), PINTO's rationalizing module is a frozen pretrained LM that uses only a few question-answer-rationale demonstrations as a prompt ( §3.1). Second, given q, a i ∈ A, and r i ∈ R, the reasoning module outputs plausibility score ρ(q, a i , r i ) Inference Figure 2: Overview of PINTO.
(1) A frozen medium-scale LM is prompted to generate choicespecific rationales.
(2) A small-scale LM is fine-tuned to reason over the generated rationales.
(3) We introduce counterfactual regularization in addition to standard training loss to ensure the rationales are leveraged properly. During inference, the rationalizing LM is prompted with a new question to generate rationales, which are provided to the reasoning module to make a prediction.
( §3.2). We also design a regularization objective that encourages the reasoning module to properly use the rationales to predict the answer ( §3.3). We describe each module in more detail below.

RATIONALIZING MODULE
Prior works mainly rely on human-annotated rationales for teaching a model to rationalize (Kumar & Talukdar, 2020;Hase et al., 2020;Sun et al., 2022). However, such rationale annotations are expensive and frequently of low quality (Aggarwal et al., 2021;Sun et al., 2022;Rajani et al., 2019), e.g., not providing sufficient knowledge to support a given answer. Meanwhile, a recent study shows that rationales automatically generated by pretrained LMs are often preferable over human-annotated rationales (Wiegreffe et al., 2021). Therefore, for PINTO's rationalizing module, we propose using a pretrained LM to generate rationales via in-context learning, which prompts the frozen LM to retrieve knowledge from its parameters (Wei et al., 2022b).
The prompt consists of a fixed set of question-answer-rationale demonstrations that are randomly selected from the training set. Each demonstration consists of a question q, answer choices A, 2 gold answer a * ∈ A, and a human-annotated free-text rationale r * ∈ R for a * (Table 1). 3 With this prompt p, we use the LM to generate rationales for every instance from the dataset. Specifically, for each a i ∈ A of some instance (q, A), the rationalizing LM's input is constructed as [p, q, A, a i ]. Then, we use greedy decoding of the LM output to obtain rationale r i for a i . Note that the LM input does not have any information about the gold answer a * . Our rationalizing module's design assumes that r i will be aligned with accurate knowledge if and only if a i = a * , since it should intuitively be difficult to retrieve correct knowledge that supports an incorrect answer choice. The reasoning module then predicts the correct answer by reasoning over the rationales for each answer choice.

REASONING MODULE
Given a question q, the answer choices A, answer candidate a i ∈ A, and rationale r i , the reasoning module learns to output plausibility score ρ i = ρ(q, A, a i , r i ). Following prior works, we use a text-to-text Transformer LM as the backbone of our reasoning module (Wiegreffe et al., 2020;Hase et al., 2020). For each a i , the reasoning module's input is defined as the token sequence where ⊕ denotes concatenation. Meanwhile, the reasoning module's 2 We include the answer choices A in the prompt so that the LM is aware of all the available choices and thus could generate a rationale that is more distinctive.
3 As opposed to full human annotation, we only need a few (usually < 8) examples per dataset. A: The answer is using less resources. Conserving resources has a positive impact on the environment. Use of resources affects the environment such as pollution.
output is obtained by sequentially teacher-forcing a i 's tokens ] into the decoder, rather than via greedy decoding. This way, we can compute the reasoning module's output token probabilities for arbitrary answer choices a i . Following Shwartz et al. (2020), we compute a i 's plausibility score ρ i by aggregating the probabilities P of tokens t j i as: Next, we use the softmax function to normalize ρ i as probability P (a i | q, A, R) = e ρi / |A| j=1 e ρj . During inference, given question q and answer choices A, the rationalizing module first generates rationales R = {r i }, then the reasoning module computes the predicted answer choice asâ = arg max ai∈A P (a i | q, A, R).

TRAINING
For multi-choice QA, the standard training objective is to maximize the likelihood of the correct answer choice using cross-entropy loss, computed as: where Q(a i | q, A) is 1 if a i = a * and 0 otherwise. Let Q(A | q, A) be the one-hot target distribution over all a i ∈ A. There can be spurious correlations between q and A (Branco et al., 2021), so the reasoning module may take undesirable shortcuts instead of properly using the rationale to predict the answer (Gururangan et al., 2018;McCoy et al., 2019). In this case, the rationales would be unfaithful in explaining the model's behavior and useless for model debugging.
To address this, we introduce a counterfactual regularization objective in which the reasoning module is regularized to output less confident predictions when the rationale is not utilized properly (i.e., shortcuts are used). This is implemented using label smoothing (Szegedy et al., 2016), which softens the target distribution Q(A | q, A) by linearly combining it with a noisy distribution U (A | q, A), often set as the uniform distribution. Therefore, given tunable label smoothing factor 0 < < 1, we compute the label-smoothed target distribution as: In order to simulate shortcut reasoning, we consider two strategies for perturbing the generated rationales r i . Token Masking addresses the case where the reasoning module ignores the rationale and instead exploits spurious cues in the rest of the input. To simulate this, we mask out the rationales in the input. Recall that the backbone of the reasoning module is a Transformer LM, which uses a self-attention mechanism to aggregate information across tokens. Hence, we implement rationale masking by zeroing the attention mask for rationale tokens. 4 Token Replacement addresses the case where the reasoning module misunderstands the rationales' meaning and thus uses them improperly. To simulate this, we randomly replace k% of the rationale tokens with other tokens uniformly sampled from the entire language vocabulary. At each fine-tuning step, we randomly select one of the strategies for obtaining perturbed rationales R = {r i }, which helps keep the LM from overfitting to any particular strategy. Then, the counterfactual regularization loss is computed as: This counterfactual regularization teaches the reasoning module to be less confident when the rationales are either absent or problematic, so that it can learn to make sounder use of the rationales.

Questions and Hypotheses
We design experiments to answer the following questions: (1) What is the impact of PINTO on task performance and rationale faithfulness? We expect PINTO's prompted rationalization and counterfactual regularization to achieve improvements in both aspects.
(2) How does the quality of PINTO's generated rationales affect its task performance? Since the generated rationales are a core component of PINTO's reasoning process, we hypothesize that PINTO's rationale generation quality is positively correlated with its task performance.
(3) How does faithfulness in rationalizing affect generalization in reasoning? We expect PINTO's counterfactual regularization to both yield more faithful rationales (i.e., higher consistency between rationale and predicted answer) and reduce reliance on spurious shortcuts, which should improve generalization in low-resource and OOD settings.
Datasets We experiment with several language reasoning datasets, each requiring the use of implicit knowledge.
(1) CSQA (Talmor et al., 2018) is a five-choice QA dataset that tests general commonsense reasoning about the concepts from ConceptNet (Speer et al., 2017). (2) StrategyQA (Geva et al., 2021) is a binary (yes/no) QA dataset in which reaching the correct answer involves inferring an underlying multi-step reasoning strategy.
(3) OBQA (Mihaylov et al., 2018) is a four-choice QA dataset that entails reasoning over both scientific texts and broad commonsense knowledge. (4) QASC (Khot et al., 2020) is an eight-choice QA dataset where questions must be answered by using common sense to compose a set of basic facts. Since the test labels for these datasets are not publicly available, we treat the official development set as our test set, while randomly splitting the official training set into a new training set and development set.

Evaluation Metrics
To evaluate the reasoning model's task performance, we use the accuracy metric and consider both ID and OOD test sets in our experiments. ID/OOD test sets are taken from the same/different dataset as the training set. To evaluate the faithfulness of the generated rationale to the reasoning model's predicted label, we adopt the LAS metric (Hase et al., 2020). LAS measures rationale-label consistency as how well the rationale helps a simulator model predict the reasoning model's predicted label. Following Hase et al. (2020), we implement the simulator as a fine-tuned T5-Base LM (Raffel et al., 2020a). To aggregate accuracy and LAS as a single metric, we use Normalized Relative Gain (NRG) metric (Chan et al., 2022b). Across all compared methods, NRG first normalizes each of the two constituent metrics' scores as values in [0, 1], then obtains the aggregate score by taking the mean of the two normalized scores.
Implementation Details For the rationalizing module, we use GPT-NeoX (Black et al., 2022), a pretrained, autoregressive, decoder-only LM with 20B parameters. To create each dataset's prompt, we manually annotate gold rationales for seven training examples. For the reasoning module, we use T5-Base (Raffel et al., 2020a), an encoder-decoder LM which has only 220M parameters (i.e., two orders of magnitude smaller than the rationalizing module). We also experiment with RoBERTa-Large (Liu et al., 2019), an encoder-only LM with 355M parameters, as the reasoning module. For RoBERTa-Large (unlike for T5-Base), we obtain the plausibility score for each choice by directly feeding the final encoder representation into a linear layer followed by a softmax classifier.
During fine-tuning, the standard training loss (Eq. 1) and our counterfactual training loss (Eq. 2) are directly combined as the overall training loss. For counterfactual training, the rationale perturbation strategy is uniformly selected (between token masking and token replacement) per training batch. For token replacement, the replacement rate is empirically set to k = 30%. All reported results are averaged over four fixed random seeds.
Baselines (1) Without Rationales is a T5-based model fine-tuned on the task dataset without using any rationales as additional input.
(2) Prompted Self-Rationalization is a GPT-NeoX LM that learns from a few examples in the prompt to firstly generate a few short sentences as the rationale and then predict the answer. Here, we use the chain-of-thought prompting configuration from Wei et al. (2022b).
(3) Distilled Self-Rationalization is a small LM (T5-base) trained on the rationales generated by the Prompted Self-Rationalization model. We implement two variants of the distillation model: a) Rationalize-First, which firstly generates the rationale and then predicts the answer, and b) Predict-First, which firstly predicts the answer and then generates the rationale. (4) NILE (Kumar & Talukdar, 2020) trains a rationalization module by fine-tuning a T5-3B model (Raffel et al., 2020a) with the rationales annotated by humans, then trains a reasoning module by fine-tuning a T5-Base model with the task dataset as in our method. We only apply NILE on the CSQA and StrategyQA datasets, since they provide human-annotated gold rationales. (5) Standard Training uses the same rationalize-then-reason pipeline as our method, except the reasoning module is not fine-tuned with the counterfactual training loss. (6) Dropout Context is the same as the Standard Training baseline, except the question is randomly dropped out from the input while fine-tuning the reasoning module. This is a strategy used in prior work to encourage the reasoning module to make good use of the input rationales (Hase et al., 2020).
Also, we consider two additional variants of PINTO as baselines: Token Masking Only and Token Replacement Only. These baselines either only use token masking or only use token replacement, respectively, to perturb rationale tokens. For all the compared models, we search the optimal hyperparameters based on the average of accuracy and LAS over the development set, where the learning rate is chosen from {1e−5, 3e−5, 1e−4, 3e−4}, the batch size is chosen from {8, 16, 32, 64}.

In-Distribution (ID) Performance
We first evaluate all methods on ID test sets. Table 2 shows the task performance of these methods, with fine-tuning methods using T5-Base as the reasoning module. We have the following two observations. First, the Prompted Self-Rationalization baseline (using the 20B-parameter GPT-NeoX) generally does not outperform the fine-tuning methods while the GPT-3 version is reported to achieve 73.50 and 66.53 in accuracy on CSQA and StrategyQA, respectively (Wei et al., 2022b). This validates that Prompted Self-Rationalization requires very large LMs to work effectively (Wei et al., 2022a). Second, simply augmenting the reasoning module with rationales (as in Standard Training) does not always lead to better results compared with the Without Rationales baseline since the rationales may not be properly utilized. The Dropout Context baseline helps to address this issue in some, but not all cases, while PINTO consistently yields the best accuracy in most of the cases. We have similar observations from results using RoBERTa-Large as the reasoning module (Table 6 of §A.1). This demonstrates the effectiveness of our counterfactual regularization method in improving ID generalization.

Out-of-Distribution (OOD) Performance
To further demonstrate the generalizability brought by faithful reasoning over rationales, we also investigate the performance of our method on OOD test sets. The intuition is that by utilizing rationales faithfully rather than fitting only the ID training data, our model achieves better OOD generalization without any fine-tuning. Table 3 shows the OOD performance of all the fine-tuning methods using T5-Base. We conclude that rationales are helpful in improving the generalizability of the model to a dataset unseen during fine-tuning. Among all the methods utilizing rationales, our method yields the best OOD performance, which confirms the benefit of faithful reasoning. A consistent conclusion can be made from the results based on RoBERTa-Large (Table 7 of §A.1). Table 2 also reports the faithfulness of all the methods involving rationalization measured by LAS. We observe that PINTO achieves a much higher score compared with the baselines except on OpenBookQA. This demonstrates that counterfactual regularization helps the reasoning module make predictions more faithfully with respect to the rationales.

PERFORMANCE ANALYSIS
How do different perturbation strategies contribute to the overall performance? Table 2 shows the results of the ablation study where we only conduct Token Masking or Token Replacement when perturbing the rationale tokens. From more cases, we note that Token Replacement leads to both better accuracy and faithfulness compared with Token Masking. This is because Token Replacement perturbs the semantics of the rationales more severely, thus further forcing the reasoning module to properly make use of the rationales. Our method yields the best results when both types of perturbation are conducted, which validates that these two strategies consider comprehensively the different ways in which a reasoning module could use the rationales improperly.  How does the smoothing factor affect accuracy and faithfulness? Intuitively, the smoothing factor in §3.3 controls how much we want to enforce the reasoning module to consider the ratio-nales. To understand its role in affecting task accuracy and rationales faithfulness, we conduct an analysis where we vary the from 0.1 to 1.0 when fine-tuning the reasoning module. Figure 4 shows accuracy on the left axis and faithfulness on the right axis. We can see that with a small (0.1), our method strikes a balance between obtaining high accuracy and ensuring faithfulness. When = 0.8, our method yields the highest faithfulness measure at the cost of accuracy. Note that there is no clear trend in the results plot due to the randomness brought by the training dynamics.

Standard
Dropout PINTO 56 58 60 62 64 Accuracy (%) Generated Generated+Annotated Figure 6: Rationale Quality Analysis. Accuracy of models with both generated and annotated rationales vs. models using only generated rationales on CSQA.
Can faithful rationales lead to better lowresource performance?
We also investigate whether, with counterfactual training, the reasoning module can be fine-tuned with less training data. Figure 5 shows the accuracy of all the fine-tuning methods. We can observe that our method consistently outperforms the baselines at different percentages of training data. The observed larger performance gap is larger when less training data is used, demonstrating the data efficiency of our method.
Can we refine the reasoning behavior via rationales? One important application of faithful reasoning is that rationales provide a way to refine the behavior of a model, i.e., we can correct reasoning mistakes by providing a better rationale. To verify this, we make use of ECQA (Aggarwal et al., 2021) which augments CSQA with human-annotated rationales. We directly provide the human-annotated rationales to the fine-tuned reasoning modules to obtain its oracle results, shown in Figure 6. We see that human-annotated rationales generally lead to performance gain for all fine-tuning methods whereof the gain of our method is the largest. This again showcases the merits of ensuring the faithful reasoning on rationales in refining a system. Is our method more sensitive to perturbed rationales? Intuitively, higher rationale faithfulness (i.e., stronger connection between the rationale the and reasoning module's behavior) should yield greater sensitivity to noisily perturbed rationales. In other words, higher performance drop (sensitivity) signals higher faithfulness. To verify this, we conduct a stress test. We choose CSQA and OpenBookQA and replace each question in the testing set with a randomly sampled question but still keep the original answer choices. We then prompt our rationalizing module with the replaced question and the original choices to obtain a set of perturbed rationales. We finally provide the perturbed rationales to the reasoning module. Our results in Table 4 show that PINTO achieves a significantly higher performance drop than the other two methods (esp. on OBQA), indicating that counterfactual regularization is effective in improving rationale faithfulness.

CASE STUDY
We provide concrete examples in Table 5 to showcase how our prompted LM rationalizes for correct and incorrect choices and how PINTO reasons more faithfully compared with the Standard baseline.
In the question (second row) from CSQA, we can see that for incorrect choices, the generated rationales do not support them to be the answer while the one for the correct choice refrigerator does.
In the question (third row) from StrategyQA, the rationale for the correct choice yes is sound and reasonable while the rationale for the incorrect choice no is factually correct but does not answer the question directly (died in a plane crash vs. died in the space journey). For both questions, PINTO properly leverages the rationales and make the correct predictions while the Standard baseline fails.

RELATED WORK
Extensive work has been done on solving implicit reasoning tasks by augmenting reasoning LMs with external knowledge beyond the task input. Prior works have explored retrieving implicit knowledge from: (1)   oven refrigerator (a) quandry: Quandry is a place where you put things that you don't know ... (b) refrigerator: Refrigerator is used to store food. (c) oven: Oven is used to bake cakes.
Q: Has every astronaut survived their space journey? yes no yes: Astronauts have died in space. For example, the Challenger astronauts died in the Challenger space shuttle explosion. no: The first astronaut to die was Yuri Gagarin. Gagarin died in a plane crash. 2021; Raman et al., 2021), (2) web corpora (Lv et al., 2020;Chen et al., 2017;Yang et al., 2015;Ryu et al., 2014), or (3) pretrained LMs (Shwartz et al., 2020;Liu et al., 2021;Bosselut et al., 2019;Shin et al., 2020). Although knowledge retrieval has shown to be helpful in boosting reasoning LMs' task performance, it may not necessarily explain the decisions made by the LM. Given the lack of transparency in neural LMs' complex behavior (Rudin, 2019;Caruana, 2019), model explainability is important for promoting human trust in NLP systems for high-stakes decision-making (Doshi-Velez & Kim, 2017;Lipton, 2018;Bender et al., 2021). We focus on rationale generation in this work as a way to both improve an LM's task performance and provide justification for its predictions.
Much of the LM explainability literature has focused on studying extractive rationales, which use input token scoring to highlight the tokens that were most important for the LM's prediction. This can be done via attention weights (Pruthi et al., 2020;Stacey et al., 2022;Wiegreffe & Pinter, 2019), input gradients (Sundararajan et al., 2017;Lundberg & Lee, 2017;Denil et al., 2014;Li et al., 2015),  Situ et al. (2021). However, extractive rationales alone are insufficient for explaining LMs' decisions on implicit reasoning tasks, since the knowledge required for such tasks is often not even part of the input. In contrast, free-text rationales aim to explain LM behavior using natural language, which is a more flexible and human-accessible to make the LM's implicit reasoning process more explicit (Narang et al., 2020;Wei et al., 2022b;Marasović et al., 2022;Zelikman et al., 2022;Rajani et al., 2019;Camburu et al., 2018;Kumar & Talukdar, 2020;Wiegreffe et al., 2020;Hase et al., 2020;Chan et al., 2022a).
Prior works on free-text rationale generation can be grouped into three paradigms. In the fine-tuned self-rationalizing paradigm, a single LM is fine-tuned to jointly generate the task output and rationale (Narang et al., 2020;Marasović et al., 2022;Zelikman et al., 2022). Since the LM parameters are shared across two relatively dissimilar objectives, they often perform worse than non-rationalizing LMs (Wiegreffe et al., 2020;Narang et al., 2020). Notably, this paradigm requires expensive rationale annotations for all training instances. In the prompted self-rationalizing paradigm, a single LM is instead frozen and prompted to jointly generate the task output and rationale, with the prompt consisting of a few input-output-rationale demonstrations (Wei et al., 2022b). This paradigm performs well and only needs a few rationale annotations for the prompt, but it is computationally prohibitive since it generally requires very large-scale LMs to work effectively (Lampinen et al., 2022;Wei et al., 2022b). In the pipeline-rationalizing paradigm, a fine-tuned rationalizing LM first generates the rationale, which is then used as input for a separate fine-tuned reasoning LM to generate the output (Kumar & Talukdar, 2020;Rajani et al., 2019). Here, the generated rationale forms a discrete (i.e., non-differentiable) bottleneck between the two modules, which complicates end-to-end training and can hurt task performance (Wiegreffe et al., 2020;Hase et al., 2020). Additionally, the dedicated rationalizing LM requires extra rationale annotation/computation costs. Moreover, none of these paradigms has a mechanism for regularizing the rationale generation to faithfully reflect the reasoning process of the LM, without hurting task performance. PINTO avoids these limitations by rationalizing via prompt-based learning (using a frozen medium-scale LM), then reasoning over the task input and rationale via counterfactual regularization (using a fine-tuned small-scale LM).

CONCLUSION
This paper presents PINTO, an LM pipeline that rationalizes with prompt-based learning and reasons via counterfactual regularization. Through prompting, we remove the need for expensive human annotation and leverage the massive knowledge encoded in a medium-scale LM to perform rationalization. With counterfactual regularization, our reasoning module learns to reason over the generated rationales more faithfully. Our experiments show that our method outperforms baselines on both ID and OOD datasets in accuracy. Also, we find that rationales are utilized more faithfully by PINTO than by other methods, leading to better task performance in low-resource settings. Furthermore, we show that PINTO's counterfactual regularization improves the reasoning module's robustness to noise in the rationalizing module's generated rationales.
Sarah Wiegreffe, Ana Marasović, and Noah A Smith. Measuring association between labels and free-text rationales. arXiv preprint arXiv:2010.12762, 2020.     Table 8 shows the results of the Pipeline approaches using LMs with different model sizes as the rationalizing module. We conducted a human evaluation of 100 generated rationales from the CSQA dataset. The evaluation is a head-to-head comparison between the human-annotated rationales and the machinegenerated rationales. Annotators were asked to judge for 5 dimensions on a 3-point Likert scale following a prior work (Wiegreffe et al., 2021): 1) Factuality (How factual is this Explanation?) 2) Grammaticality (Is this Explanation grammatical?) 3) New Info (Does the Explanation provide new facts, information, or reasoning not stated in the Question and Answer?) 4) Supports Answer (Is the Explanation relevant to justifying the Answer?) 5) Completeness (Does the Explanation provide enough information to jusify the answer?) We obtain a fair level of agreement measured by Fleiss Kappa (k=0.34) for the evaluation. The results in Table 9 show that machine-generated results are competitive with human annotation on most of the evaluating dimensions. Generated rationales are even judged to be more grammatical than human annotations. As for completeness, generated rationales are slightly worse than human annotations. We think this is because the human annotators were explicitly encouraged to provide more comprehensive rationales when annotating the CSQA dataset (Aggarwal et al., 2021).    Q: When standing miles away from Mount Rushmore Answer Choices: (a) the mountains seem very close (b) the mountains are boring (c) the mountains look the same as from up close (d) the mountains seem smaller than in photographs A: The answer is the mountains seem smaller than in photographs.When an object is far away, it takes up less of your field of view, and so seems smaller than in the photographs.
Q: When food is reduced in the stomach Answer Choices: (a) the mind needs time to digest (b) take a second to digest what I said (c) nutrients are being deconstructed (d) reader's digest is a body of works A: The answer is nutrients are being deconstructed. The stomach is part of the digestive system. The breaking down of food into nutrients occurs in the digestive system.
Q: Poison causes harm to which of the following? Answer Choices: (a) a Tree (b) a robot (c) a house (d) a car A: The answer is a Tree.A tree is a living thing. Poison causes harm to living things.
Q: A magnet will stick to Answer Choices: (a) a belt buckle (b) a wooden table (c) a plastic cup (d) a paper plate A: The answer is a belt buckle. A belt buckle is made of metal. If a magnet is attracted to a metal then that magnet will stick to that metal.
Q: Deer are less safe in the woods because wolves Answer Choices: (a) have fur (b) howl (c) have claws (d) have tails A: The answer is have claws. Claws are used by wolves to catch prey like deer.
Q: An electric car causes Answer Choices: (a) more CO2 emissions (b) equal CO2 emissions (c) electric emissions (d) less CO2 emissions A: The answer is less CO2 emissions. An electric car uses less gasoline than a regular car and thus causes less CO2 emissions.  Khot et al. (2020) as the rationalization.
Q: How do you reduce pollution? Answer choices: (a) igniting fuel and oxidiser (b) transportation technology (c) wasting (d) not recycling (e) burning fossil fuels (f) converting electricity to heat (g) water conservation (h) using less resources A: The answer is using less resources. Conserving resources has a positive impact on the environment. Use of resources affects the environment such as pollution.
Q: what will move to another area if their habitat will no longer support them? Answer choices: (a) density (b) Birds (c) squids (d) humans (e) clouds (f) gravity (g) cows (h) Whales A: The answer is cows. If a habitat can no longer support animals then those animals will move to another area. Cows are social animals.

) glass (h) eyes
A: The answer is coal. If an object is black then that object absorbs all visible light. Light grains are quartz, Black grains are coal.