{'fraction_non_alphanumeric': 0.04024635387707968, 'fraction_numerical': 0.02437876908601001, 'mean_word_length': 4.810636182902584, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A New Paradigm for Evaluating Avoidance/Escape Motivation

Abstract Background: Organisms have evolved to approach pleasurable opportunities and to avoid or escape from aversive experiences. These 2 distinct motivations are referred to as approach and avoidance/escape motivations and are both considered vital for survival. Despite several recent advances in understanding the neurobiology of motivation, most studies addressed approach but not avoidance/escape motivation. Here we develop a new experimental paradigm to quantify avoidance/escape motivation and examine the pharmacological validity. Methods: We set up an avoidance variable ratio 5 task in which mice were required to press a lever for variable times to avoid an upcoming aversive stimulus (foot shock) or to escape the ongoing aversive event if they failed to avoid it. We i.p. injected ketamine (0, 1, or 5 mg/kg) or buspirone (0, 5, or 10 mg/kg) 20 or 30 minutes before the behavioral task to see if ketamine enhanced avoidance/escape behavior and buspirone diminished it as previously reported. Results: We found that the performance on the avoidance variable ratio 5 task was sensitive to the intensity of the aversive stimulus. Treatment with ketamine increased while that with buspirone decreased the probability of avoidance from an aversive stimulus in the variable ratio 5 task, being consistent with previous reports. Conclusion: Our new paradigm will prove useful for quantifying avoidance/escape motivation and will contribute to a more comprehensive understanding of motivation.


Introduction
Motivation drives us to execute actions and provides the vigor needed to overcome obstacles and achieve goals. Decades of research have led to the recognition that motivated behavior is complex, consisting of multiple interacting components that can be dissociated at both the behavioral and neural levels (Salamone and Correa, 2002;Robinson and Berridge, 2013;Kelley, 2004;Natsubori et al., 2017;Tsutsui-Kimura et al., 2017). According to a widely accepted view, motivation can be divided into 2 subcategories: reward gain-based and aversion avoidance/ escape-based motivation (Elliot, 1999;Elliot and Thrash, 2002;Campese et al., 2015). These 2 subcategories differ as a function of valence: in approach motivation, behavior is instigated or directed by a positive/desirable event or possibility; in avoidance/escape motivation, behavior is instigated or directed by a negative/undesirable predicted or actual event. Despite several recent advances in understanding the neurobiology of motivation (Salamone and Correa, 2002;Tsutsui-Kimura et al., 2017), only a few studies have addressed avoidance/escape motivation (Salamone, 1994;Perrotti et al., 2013). A more comprehensive understanding of motivation would be possible with a strategy of implementing behavioral assays that can assess avoidance/escape motivation.
Evaluating the latter type of motivation with existing behavioral assays has been a challenge. For instance, the active avoidance test with 2 shuttle boxes, so-called 2-way active avoidance, is one of the widely used animal models (Da Cunha et al., 2009;Boschen et al., 2011). In this task, the animal learns to predict the occurrence of an aversive event (typically a foot shock) based on the presentation of a specific stimulus (typically a tone) to avoid the aversive event by moving to a different compartment. The number of avoidances (the number of crossings to the other compartment during the warning signal) is taken as an index of avoidance motivation. The number of escapes (the number of crossings to the other compartment during receiving the aversive event) is taken as an index of escape motivation. However, these dependent variables can easily reach a ceiling level, making them useless in assessing motivation. The main reason for this issue resides in the inflexibility to modulate the amount of effort required to avoid or escape the aversive event.
Here we developed a new strategy that overcomes this limitation and enables us to quantify avoidance/escape motivation. In our new paradigm, mice were required to press a lever, instead of shuttling the box, for avoiding or escaping from an aversive event. As the task requires a variable number of lever presses, it will dampen the response-outcome association and will prevent their performances from reaching ceiling levels. We called this task the avoidance variable ratio 5 (VR-5) task. We first validated the avoidance VR task as a measure of avoidance/escape motivation by showing that it is sensitive to the magnitude of the aversive stimulus. Next, to evaluate the effectiveness of this strategy, we looked at how performance on the avoidance VR-5 task was affected by lower doses of ketamine (KET; 1 or 5 mg/kg), which is known to induce avoidance/escape behaviors in helpless animals (Maeng et al., 2008;Belujon and Grace, 2014), and by the doses of buspirone hydrochloride (BUS; 5 or 10 mg/kg), which is known to reduce avoidance/escape behaviors (Liang et al., 1998).

Ethical Statement
All animal procedures were conducted in accordance with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and approved by the Animal Research Committee of Keio University.

Animals
Subjects were 19 129SvEvTac male mice older than 120 days of age and weighing 26 to 30 g at the start of the experiment. This strain is known to show stable performance on operant tasks (Thomsen and Caine, 2006;Haluk and Wickman, 2010). They were food deprived to motivate them to earn dustless precision pellets (Bio-serv) as a reward. Thereafter, their body weight was maintained at 85% of free-feeding weight. Food restriction was conducted only during shaping-1 and -2 in the training phase. To control the food deprivation level equally in each animal, the animals were housed individually. Water was available ad libitum in home cages throughout the entire experiment. Subjects were maintained on a 12-hour-light/-dark schedule (lights on 8 am) and tested during the light phase. The sample size employed in this study was determined according to previous reports (Liang et al, 1998;Maeng et al., 2008;Belujon and Grace, 2014).

Apparatus
Aluminum operant chambers measuring W22 × D26 × H18 cm (Med Associates Inc.) were used. It was equipped with a food magazine located on a wall of the chamber, 2 retractable levers, and one speaker on either side of the food magazine. The apparatus was controlled by a computer program written in MED-PC language (Med Associates Inc.). As the behavioral parameters were automatically collected by the PC system, any possible bias from experimenters should have been avoided.

Behavioral Procedures
Training Food-restricted mice were initially trained to press the lever to earn a food pellet (shaping-1, Figure 1A). A trial started with the 2 levers presentation. Only one lever is designated as "active" (triggering delivery of food reward) and the right/left allocation of active lever was counterbalanced between mice. A single active lever press resulted in food reward delivery and retraction of both levers. Then, 60 seconds of inter-trial interval was added, followed by the automatic starting of the next trial.
Following 2 successive sessions of obtaining ≧50 pellets, the schedule moved to shaping-2 in which an alarm tone (100 dB, 1900 Hz, 0.5 seconds intermittently) was presented simultaneously with the lever presentation. A single active lever press resulted in the retraction of levers, the termination of the alarm tone, and the delivery of the food reward. Following 5 to 10 days of daily shaping-2 sessions, the schedule moved to the next step.
Then, the alarm tone was paired with an aversive stimulus (conditioning, Figure 1A). A trial was started with the alarm tone presentation without lever presentation. After 60 seconds of the alarm tone exposure, mice received a foot shock (0.15 mA, 10 seconds continuously), followed by the termination of the foot shock and the alarm tone. Each trial was divided by a 60-second inter-trial interval. Conditioning consisted of only one session (20 trials).
On the following day, mice received avoidance FR-1 ( Figure 1A), in which an alarm tone (100 dB, 1900 Hz, 0.5 seconds

Significance Statement
Evaluating the avoidance/escape motivation with existing behavioral assays has been a challenge, since they are inflexible to modulate the amount of effort required to avoid or escape the aversive event. We developed a new behavioral paradigm, an avoidance variable ratio 5 task, which overcomes this limitation and enables researchers to quantify avoidance/escape motivation. In this task, mice were required to press a lever for variable times to avoid a subsequent aversive stimulus or to escape the aversive event if mice failed to avoid it. We confirmed the pharmacological validity of this paradigm by using ketamine and buspirone. Our new paradigm will contribute to the development of effective therapeutic approaches to psychiatric disorders associated with maladaptive avoidance/escape responses.
intermittently) was presented simultaneously with the lever presentation (warning period, 60 seconds). A single active lever press during the warning period resulted in the start of a 30-second safety period (retraction of the levers and termination of the alarm tone) without delivery of food reward. This behavior was counted as an avoidance lever press and regarded as an index of avoidance motivation. If the mice failed to press the active lever during the warning period, they received 10 seconds of foot shock (shock period). If they pressed the active lever during the shock period, it resulted in the start of a safety period. If the mice failed to both avoid and escape from the shock, such trial was regarded as an omission. The number of magazine entries was also counted to address the level of food seeking.
All the training sessions were conducted for 60 minutes per session and 1 session per day. Food reward was delivered only in the shaping-1 and shaping-2 trainings.  figure) press was reinforced by a palatable pellet. The alarm tone was synchronously presented with levers and terminated after the active lever press in shaping-2. In conditioning, the alarm tone was paired with an aversive stimulus 20 times. Mice were required to press the active lever during alarm tone presentation in the avoidance FR-1 task. (B) Testing procedure. Mice had to press the active lever variable times (1, 3, 5, 7, or 9) to avoid or escape from the foot shock during the warning or shock periods, respectively.

Test
Following 2 successive sessions of obtaining ≥50 avoidance lever presses in the avoidance FR-1, the schedule moved to avoidance variable ratio-5 (VR-5). In this task, the mice had to press the active lever variable times (1, 3, 5, 7, or 9) to avoid or escape during the warning or shock periods, respectively. The number of active lever presses during the warning period was regarded as an index of avoidance motivation, and that within the foot shock period was regarded as an index of escape motivation. The other procedures were equal to that of avoidance FR-1 ( Figure 1B).
Physiological (variable shock intensity) and pharmacological (KET or BUS injection) manipulations were conducted after the performance on the VR-5 schedule of reinforcement was considered stable when the percentage of omissions in a session deviated by ≤10% for at least 3 consecutive days.
Test sessions were conducted for 60 min per session and 1 session per day.

Parameters
We used 9 behavioral parameters described as follows: (a) Total trials completed (count per session) (b) %Avoidance: (the number of avoidance trials/total trial number) × 100 (c) %Escape: (the number of escape trials/total trial number) × 100 (d) %Omissions: (total trial number -the number of avoidance -the number of escape /total trial number) × 100 (e) First lever press latency(s): the mean time from the trial start to the first active lever press (f) Time spent to complete the VR(s): the mean time elapsed from the first active lever press to the achievement of the required number of active lever presses (g) Inactive lever presses (count per session) (h) Magazine entries (count per session) (i) Immobility(s): time spent in immobile behavior manually counted

Stimulus Intensity in the Avoidance VR-5 Task
We initially addressed the minimum shock intensity to induce immobility using 5 naïve mice; 0.15 mA is the minimum intensity to induce immobility in all mice tested. We then examined the effect of the shock intensity in the VR-5 schedule from 0.15 to 0.35 mA with a 0.05-mA increment using 8 mice. The order of application of varied shock intensities was counterbalanced. Data were averaged over the 2 days of testing at each intensity.
Nine mice were used in this pharmacological study. Each mouse was injected with both drugs at 3 different dosages (zero, low, high). The orders of injections and drugs were counterbalanced. Intervals between different dosages were more than 2 days and intervals between different drugs were more than 7 days. This study lasted for about 5 weeks and the VR-5 task was conducted on weekdays.

Data Analysis
For all experiments including basal performance, physiological validity, and pharmacological studies, each behavioral parameter was analyzed separately using a repeated-measures ANOVA. Multiple comparisons test using the Bonferroni's method was performed when a significant main effect was observed.
The alpha level was set to .05. All statistical procedures were conducted using SPSS (version 20).

Basal Performance on the Avoidance VR-5 Task
The median number of training sessions required for each stage was 6 (minimum 4 and maximum 9), 7 (minimum 5 and maximum 9), and 5 (minimum 3 and maximum 9) for shaping-1, shaping-2, and avoidance FR-1, respectively. We trained 19 animals and 17 of them were used in the experiments (2 animals failed to achieve the avoidance VR-5 task). After completion of the training phases, subjects were tested on the avoidance VR-5 task ( Figure 1B). The number of magazine entry was significantly decreased after the conditioning phase, indicating that the reinforcing property of the food reward was diminished in the avoidance task (n = 8, main effect of task types: F 4, 28 = 7.32, P < .01; multiple comparisons of Bonferroni method: P < .05 for avoidance FR-1 vs. Shaping-1 or -2, and avoidance VR-5 vs. others, Figure 2A).

Physiological Validity of the Avoidance VR-5 Task
To validate that the avoidance VR-5 task was sensitive to differences in motivation, we manipulated the level of a parameter known to alter motivated responding: the level of shock intensity. Performances in the avoidance, but not escape, VR-5 task were affected by the different levels of foot shock intensity (n = 8, main effect of intensity, %avoidance: F 4, 28 = 8.02, P < .01; %escape: F 4, 28 = 0.43, P = .79; %omissions: F 4, 28 = 10.31, P < .01) ( Figure 3A-C). Multiple comparisons of Bonferroni's method revealed that subjects were the most motivated to work for avoiding the 0.2-mA foot shock as they made more lever presses for avoiding the aversive stimulus at this particular intensity (0.2 mA vs other intensities: P < .05). Percent omissions were also significantly decreased with a 0.2-mA foot shock compared with greater intensities (P < .05). Accordingly, immobile duration was significantly prolonged with higher shock intensity (main effect of intensity, F 4, 28 = 25.89, P < .001), resulting in impaired avoidance motivation. In turn, 0.2-mA shock intensity would not induce conditioned fear responses that precludes an evaluation of motivated behavior. Thus, the goal-oriented avoidance behavior was sensitive to the aversive stimulus magnitude and the optimal foot shock intensity inducing avoidance behavior was 0.2 mA.

KET, but Not BUS, Leads to Greater Persistence in the Avoidance VR-5 Task
A lower dose of KET has been reported to ameliorate impaired avoidance and/or escape responses in rats undergoing helplessness assessments (Maeng et al., 2008;Belujon and Grace, 2014). We thus tested mice in FR-1 and VR-5 schedules of avoidance to determine whether KET would lead to an increase in avoidance and/or escape performances.
We also tested the effects of BUS, an anxiolytic psychotropic drug, on the performances in a VR-5 task. Treatment with BUS significantly decreased %avoidance (n = 9, main effect of doses, F 2,16 = 5.204, P < .05, Figure 4G) and increased %omission (F 2,16 = 4.760, P < .05, Figure 4I), but did not affect %escape (F 2,16 = 2.89, P = .09; Figure 4H), the latency of the initial lever press (F 2,16 = 2.24, P = .14; Figure 4J), the time spent to complete the VR (F 2,16 = 1.91, P = .18; Figure 4K), or the number of total trials completed (F 2,16 = 2.45, P = .12; Figure 4L). Multiple comparisons of Bonferroni's method revealed that 10 mg/kg of BUS impaired avoidance motivation (P < .05), increased failure to avoid/escape from the shock, and increased immobile behavior (Supplemental Figure 2D) but did not affect the number of inactive lever press (Supplemental Figure 2E) or magazine entries (Supplemental Figure 2F). Therefore, BUS administration impaired performances in the avoidance VR-5 task as previous reported; however, we have to note that BUS induced immobile behavior.

Discussion
To evaluate avoidance/escape motivation, we combined active avoidance strategy and variable ratio schedule, and established the avoidance VR-5 task. In this task, we demonstrate that the application of optimized shock intensity is capable of increasing in goal-directed lever presses and decreasing avoidance/escape failures. To test the efficacy of our new strategy, we used drugs that were already tested in the 2-way active avoidance test (0, 1, or 5 mg/kg of KET and 0, 5, or 10 mg/kg of BUS) as a tool to address the behavioral profiles of mice during the avoidance VR-5 task. Treatment with KET led to increases in the probability of avoidance/escape and decreases of avoidance/escape failures in the task, while treatment with BUS decreased the probability of avoidance and increased avoidance/escape failures in the tasks. Our results demonstrate the efficacy of the avoidance VR-5 task to address avoidance/escape motivation in mice.
In our pilot study, we experienced that mice did not press lever to avoid the shock (n = 9, data not shown). Therefore, we employed a reward-based approach strategy to shape the lever press behavior in initial training phases (shaping-1 and -2) and conducted conditioning before the avoidance FR-1 (Figure 1). The number of magazine entries significantly decreased after moving to the avoidance tasks (Figure 2A), indicating that the mice switched the strategy from reward oriented to avoidance oriented. Although the precise mechanisms underlying such learning strategy used in this study are largely unknown, we succeeded in shaping the mice to press the lever in order to avoid the upcoming aversive stimulus.
We obtained an inverse bell-shaped intensity-response curve in the avoidance VR-5 task ( Figures 3A). This was not surprising because it is well known that the expectation of a strong aversive stimulus can bias the animal's decision more to flight rather than fight against (Crawford and Masterson, 1982). Indeed, the mice spent more time in immobility in higher stimulus intensities ( Figure 3D).
KET with moderately high doses (50-75 mg/kg) is known to have a general locomotor stimulant effect in rodents (Leite et al., 2008;Ribeiro et al., 2013). In this study, 5 mg/kg of KET increased %avoidance and decreased %omissions ( Figure 4A, C). Since the number of inactive lever presses or magazine entries were not affected by the KET treatment (Supplemental Figure 2B,C), KETenhanced avoidance/escape behaviors cannot be accounted for by an increase of non-goal-directed hyperactive responses. This interpretation is consistent with previous reports using low doses of KET (1-10 mg/kg) (Ghasemi et al., 2010;Belujon and Grace, 2014). A low dose of KET is also known to have an analgesic effect in humans and animals (Kator et al., 2016;Hillhouse and Negus, 2016). The analgesic effect, however, cannot explain the KET-enhanced avoidance/escape behaviors, because if the mice became less sensitive to the aversive stimulus, the avoidance/escape behaviors should have decreased ( Figure 3A). Taken together, this evidence suggests that a low dose of KET enhanced avoidance/escape motivation in our new task.
KET is known to produce fast (<24 hours) and slow (>24 hours) antidepressant effects in humans (Berman et al., 2000;Zarate et al., 2006) and rodents (Maeng et al., 2008;Belujon and Grace, 2014). In this study, the effects of KET on avoidance motivation were observed after 20 minutes of the administration ( Figure 4A) but disappeared after 24 hours (Supplemental Figure  S3), implying that KET exerts fast but not slow effects on avoidance motivation.
The neural mechanisms of KET-induced enhanced avoidance/escape motivation remain unclear. Numerous studies have proposed specific effects of dopamine (DA) on avoidance learning. Depletion of DA neurons or axons by 6-hydroxydopamine in the substantia nigra pars compacta (SNc) (Cooper et al., 1973;Jackson et al., 1977), nucleus accumbens (NAc) (Mc Cullough et al., 1993), or prefrontal cortex (PFC) (Sokolowski et al., 1994) impaired the development and maintenance of active avoidance and escape strategies. The D2 antagonist injections into NAc did not impair acquisition but reduced avoidance response, whereas D1 antagonist injections into NAc impaired both acquisition and active avoidance (Boschen et al., 2011;Wietzikoski et al., 2012). Although the precise mechanisms are not clear, systemic KET injection increases DA neuron firing in the VTA (Belujon and Grace, 2014) and the release of DA in the NAc (Moghaddam et al., 1997). Therefore, we speculate that KET-mediated enhanced mesolimbic DA activity is involved in the KET-induced increase of avoidance responses. BUS is known to decrease avoidance behaviors in the 2-way active avoidance test (Liang et al., 1998). Similar to the previous report, our study demonstrated that the high dose (10 mg/ kg) of BUS decreased avoidance and increased omission in the avoidance VR-5 task ( Figure 4G, I). As the numbers of inactive lever presses and magazine entries were intact, it is likely that the general locomotor activity was not impaired (Supplemental Figure 2E,F). Rather, the high dose of BUS induced prolonged immobile behavior (Supplemental Figure 2D). These findings suggest that the effect of BUS in our study was partly accounted for the increased conditioned fear.
Maladaptive avoidance/escape behaviors are well characterized in several psychiatric disorders. Avoidance/escape deficits are observed in anxiety and depressive disorders (e.g., avoiding social events in social phobia, avoiding places or thoughts in posttraumatic stress disorder, and helplessness in depressive patients). Recently, persistent avoidance was found to be a key feature of obsessive-compulsive disorder (Gillan et al., 2014(Gillan et al., , 2015. Our new paradigm will guide the way to develop effective therapeutic approaches to alleviate maladaptive avoidance/ escape responses observed in psychiatric disease models.