{'fraction_non_alphanumeric': 0.04234684079124256, 'fraction_numerical': 0.01080276550797004, 'mean_word_length': 4.162081784386617, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 8, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Risk Sensitive Path Integral Control

Recently path integral methods have been developed for stochastic optimal control for a wide class of models with non-linear dynamics in continuous space-time. Path integral methods find the control that minimizes the expected cost-to-go. In this paper we show that under the same assumptions, path integral methods generalize directly to risk sensitive stochastic optimal control. Here the method minimizes in expectation an exponentially weighted cost-to-go. Depending on the exponential weight, risk seeking or risk averse behaviour is obtained. We demonstrate the approach on risk sensitive stochastic optimal control problems beyond the linear-quadratic case, showing the intricate interaction of multi-modal control with risk sensitivity.


Introduction
The objective in conventional stochastic optimal control is to minimize an expected cost-to-go [10]. Risk sensitive optimal control generalizes this objective by minimizing an expected exponentiated cost-to-go. Depending on its risk parameter, expected exponentiated cost-to-go puts more emphasis on the mode of the distribution of the cost-to-go, or on its tail, and in that way allows for a modelling of more risk seeking or risk averse behaviour. The conventional optimal control can be viewed as a special case of risk sensitive optimal control with a risk neutral parameter.
Risk sensitive control was first considered by Howard and Matheson [12] in discrete space, and by Jacobson [13] in continuous space in the LEQG (Linear Exponential Quadratic Gaussian) problem, which is the risk sensitive analogue of the linear quadratic Gaussian (LQG) problem. Relations with other fields such as differential games [13,9] and robust control [11,6] have initiated a lot of interest for risk sensitive control.
The dynamic programming (DP) principle by Bellman [2] provides a well-known approach to a global solution in stochastic optimal control. In the continuous time and state setting that we will consider, it follows from the DP principle that the solution to the control problem satisfies the so-called Hamilton-Jacobi-Bellman (HJB) equation, which is a second order nonlinear partial differential equation [10,18,3]. If the dynamics is linear and the cost is quadratic in both state and control, the HJB equation can be solved exactly, both for LQG and LEQG.
Recently, a path integral formalism has been developed to solve the HJB equation. This formalism is applicable if (1) both the noise and the control are additive to the (nonlinear) dynamics, (2) the increment in cost is quadratic in the control, and (3) the noise satisfies certain additional conditions. When these conditions are met, it can be shown that the nonlinear HJB equation can be transformed into a linear partial differential equation, which can be solved by forward stochastic integration of a diffusion process [16,17]. This formalism contains LQG control as a special case. Recently, path integral optimal control has been identified as a member of a richer framework of control models, in which the cost function is written as a KL-divergence [20].
An interesting observation in [16,17] is the phenomenon of symmetry breaking in multi-modal systems, i.e, in problems where several local minima of the cost co-exist. This symmetry breaking manifests itself as a delayed choice, keeping options open and using the fact that the noise may help to come closer to one of the options at no additional cost.
Path integal methods have been applied to optimal control of collaborative multi-agent systems [23,24]. The optimal control is written as a superposition of single-agent to single-target controls, and the multiagent control problem is mapped to a graphical infer-ence problem. Similar superposition principles have been used in applications, e.g. the control of character animation [5]. In general, the path integral controls are intractable, and approximate inference methods must be applied [22].
In this paper we show how path integral control generalizes to risk sensitive control problems. The required conditions to apply path integral control in the risk sensitive case are the same as those in the risk neutral setting. As a consequence, the characteristics of path integral control, such as superposition of controls, symmetry breaking and approximate inference, carry over to the setting of risk sensitive control. Furthermore, we will make use of the path integral solutions to obtain insight in the consequence of risk sensitive control, and in particular interpret risk sensitive optimal control as emergent behaviour of an agent with an optimistic or pessimistic attitude.
We start with a review of risk sensitive control in section 2. The novel generalization of path integral control to risk sensitive control problems is presented in section 3. In sections 4 and 5 we give a more informal account on the behaviour of risk sensitive control problems with path integral control. We finish with a discussion in section 6.

RISK SENSITIVE STOCHASTIC OPTIMAL CONTROL
In this section we review risk sensitive stochastic optimal control. Risk neutral stochastic optimal control is covered as a special case. Details can be found in e.g. [3,8,14].
We consider a stochastic process X u in R d satisfying the controlled dynamics where b is the R d -valued autonomous dynamics, B is a full rank d×k matrix-valued function, k ≤ d, u is a R kvalued control, σ is a constant full rank k × k matrix, and W is a Wiener process in R k which models the noise in the system.
The performance of the system is evaluated by the cost function This finishes the proof.

THE HJB EQUATION
We give a formal derivation of the HJB equation for risk sensitive stochastic optimal control problems in continuous space and time and with a finite horizon.
By the dynamic programming principle, the optimal cost function J θ satisfies for any time r, t ≤ r ≤ T . The first two identities follow directly from the definitions. We define a function E θ by By Itô's chain rule for stochastic processes [10], where A u is the differential operator In case of zero control we simply write A for A 0 . We insert this expression for E θ (r, X u r ) in equation (2). If θ > 0, then we can drop the 1 θ log, and we find Dividing by r − t, and taking the limit r ↓ t yields Dividing by θE θ (t, x), we find the HJB equation (3) In case θ < 0, the HJB equation (3) also holds, and it is derived in a similar way.

RISK SENSITIVE PATH INTEGRAL CONTROL
The novel combination of path integral control and risk sensitive control is the subject of the present section. It generalizes risk neutral path integral control which corresponds to the case θ = 0. We consider instantaneous control cost functions L u of the form and we assume that for some λ 0 ∈ R. Substitution of L u in the HJB equation (3) yields with the optimal control given by then the HJB equation is linear, and its solution is given by the path integral due to the Feynman-Kac formula [7,15], where X satisfies the dynamics (1) without control. Otherwise, we define a logarithmic transformation of J θ by Substituting this expression for J θ in the HJB equation, and subsequently dividing by −λ θ Z −1 θ , yields The term proportional to Z −1 θ vanishes after substituting λ θ = λ 0 (1 − λ 0 θ) −1 , and the resulting HJB equation is linear. The solution of the linear HJB equation is given by the path integral The optimal value function satisfies where C is the cost function under zero control, with the limit case at θ = λ −1 0 given by equation (5). Since , we find in a way similar as in Proposition 2.1 that the optimal value function J θ is constant as a function of θ if and only if the cost C(t, X) under zero control is constant with probability one, and it is strictly increasing otherwise.
and in the limit θ → ∞ (i.e., λ −1 In particular, this shows that if either of these limits is finite, J θ (t, x) is independent of x and the optimal control will be zero.
One could argue that θ → −∞ is the extreme optimistic limit. In this limit, J θ = inf X C(t, X), which is the cost under zero control following the most optimistic path due to the noise leading to the lowest possible cost. In this extreme optimistic view, the noise will lead the agent to the most optimal cost. There is no need for control, since this will only lead to additional costs. On the other hand, θ → ∞ is the extreme pessimistic limit. J θ = sup X C(t, X), which is the cost under zero control following the most pessimistic path due to the noise leading to the highest possible cost. According to this extreme pessimistic view, any additional control is pointless, since the noise realization will be such that the worst path will be realized anyway.

LINEAR EXPONENTIAL QUADRATIC GAUSSIAN
The running example is a one-dimensional system with linear dynamics, b = 0, and zero path-costs V (t, x) = 0, B = 1, R and σ are proportional to the identity and are considered as scalars. In this section we furthermore take a quadratic end cost around a target µ, Under these assumptions, in particular the quadratic end-costs, the optimal control can be computed using LEQG (Linear Exponential Quadratic Gaussian) theory [4]. The results in this section are mainly to illustrate the path integral approach to risk sensitive control.
Since V (t, x) = 0, the expectation E t,x can be computed by a convolution with the transition probability from initial state to end state, ρ(y, T |x, t), that follows from the zero-control dynamics. In this example, this transition probability is so the path integral Z θ (t, x) follows from a Gaussian convolution with e −α 2 |x−µ| 2 /2λ θ , yielding up to a finite prefactor is not satisfied, then the integral blows up and Z θ (t, x) is infinite, otherwise, the integral is well defined, and the resulting optimal control can be computed, yielding .
Generally speaking, we see that agents with larger θ have a larger control: > 0, the amplitude of the control |u * θ (t, x)| for fixed x is zero in the limit of large time horizons T − t, and it increases over time. If θ = 1 σ 2 R 2 , so λ −1 θ = 0, the amplitude of the control |u * θ (t, x)| for fixed x is constant in time. On the other hand, if θ > 1 σ 2 R 2 , so λ −1 θ > 0, the amplitude of the control |u * θ (t, x)| is undefined if the time horizon is larger than σ 2 α 2 /(θ − λ −1 θ ), it is infinite at exactly this limit and decreases over time if the horizon is smaller than this limit. With larger horizon times, the expectation of reaching infinite costs in the tails of the quadratic function due to the noise blows up. These results are well known from LEQG control theory [4].

PIECEWISE CONSTANT END-COSTS
In this section we again consider control problems with zero path costs V (t, x) = 0. We consider piecewise constant end-costs φ(x). Thus, these control problems are not of the LEQG kind. We assume B = I for simplicity. We will show that the control with given θ can be expressed as a weighted sum of controls with θ = 0, similar to the approach in [23,24] and later more general in [21]. This expression will allow us to analyse the different behaviours for agents with different θ's.

A SINGLE REGION
First we consider the standard θ = 0 case where the agent starts in state x at time t, and the end-cost is finite in a certain region S and infinite otherwise, The path integral Z θ (t, x), also interpreted as a partition function, is then given by Z θ (t, x) = exp(−c/λ 0 )l(t, x|S), with l(t, x|S) = S ρ(y, T |x, t)dy with ρ(y, T |x, t) the transition probability from initial state to end state according to the zero-control dynamics. The θ = 0 optimal control with this cost then is u * 0 (t, x|S) = λ 0 (R ⊤ R) −1 l(t, x|S) ∂l(t, x|S) ∂x (10) which is independent of the value of c.

WEIGHTED SUM OF CONTROLS
Now we consider nonzero θ and an end-cost function in which the S i are non-overlapping sets covering the whole state space.
The partition function according to (6) results in By taking the derivative of log Z θ (t, x), the optimal control follows directly and can be written as a weighted sum of θ = 0 optimal controls:

TARGETS AND THREATS
A special case is where there are M bounded regions S i , and an unbounded remaining set S 0 = R d \ ∪S i . We may set c 0 = 0, since a global additive constant to the end-cost will not affect the control. Then there are two types of regions, those with negative costs, c i < 0, and those with positive costs, c i > 0. The former are interpreted as targets, since it is favorable to arrive there, and the latter as threats, since these are better avoided.
We rewrite the partition function as so the optimal control is now with weights .  Here we note that if λ θ ≫ 0, the control is dominated by the targets, due to the weights w i with large factors exp(−c i /λ θ ). In a similar way, if λ θ ≪ 0, the control is dominated by the threats. Due to the prefactor λ θ /λ 0 in (12), which is negative in this case, the control is directed away from the threats. This is illustrated in the first example, see Figure 1.
We consider the end cost given by equation (11) with a single small region S 1 around x = 0. Depending on the sign of c, this models a single target (c < 0) or a single threat (c > 0) in S 1 . We assume b(t, x) = 0 so that ρ is Gaussian and l(t, x|S 1 ) can be expressed in closed form using error functions. In the figure, the controls as a function of x for fixed t are plotted for different θ's, for both target and threat.
In the figure, we see indeed only significant controls in combinations of positive λ −1 θ and targets on the one hand and negative λ −1 θ and threats on the other hand. The case with λ −1 θ = 0 is somewhere in between. Furthermore, we see that the nonzero control has a bounded support, that increases if |λ −1 θ | increases. This can be understood from the fact that the control is only significantly nonzero if the weight w 1 is significantly nonzero, i.e., if the product of |(exp(−c/λ θ ) − 1)| and l(t, x|S 1 ) is significantly big. For the factor l(t, x|S 1 ), which is independent of θ, this means that the agent must get sufficiently close to the target/threat. If not, control cost towards the target is too expensive compared to the reward, or the probability to hit the threat is so small that a significant control is not needed. With larger values of −c/λ θ , the domain for which |(exp(−c/λ θ ) − 1)|l(t, x|S 1 ) and hence u * θ (t, x) is significantly nonzero is larger, as can be seen in the figure. On the other hand, with larger |λ −1 θ | the prefactor λ θ /λ 0 in (12) will be smaller, so that in the regime where |(exp(−c/λ θ ) − 1)|l(t, x|S 1 ) is large, the control decreases with larger |λ −1 θ |, as can be seen in the figure.
The second example considers the phenomenon of symmetry breaking with different values of θ and c. In the case of θ = 0 and two targets, the phenomenon of symmetry breaking in the optimal control, leading to a delayed choice, is described using the path integral formalism in [16,17]. Here we show that for other values of θ, such that cλ −1 θ ≫ 0 the symmetry breaking with two targets remains essentially the same. See Figure 2. There are two target regions of size ǫ, located at −1 and +1. The left figure shows that if T − t = 1, the optimal control is towards the middle for all values of x. The right figure shows that there is symmetry breaking at time T − t = 0.5, i.e., depending on the state x, the agent makes a choice towards which target it steers. Furthermore, we see that the point of symmetry breaking is independent of θ, although the magnitude of control does depend on θ. For instance, we see that if θ is such that λ −1 θ < 0, the control is about zero. This latter phenomenon can be understood from the fact that with such θ, the control is much less sensitive to targets, as we have discussed earlier.
The symmetry breaking for such θ where λ −1 θ is large can be understood as follows. Consider again b = 0, so that ρ is Gaussian. We model two targets at µ 1 = 1 and µ 2 = −1 of infinitesimal width ǫ. Furthermore, assume that −cλ −1 θ is sufficiently large, so that the factor exp(−cλ −1 θ )l(t, x|[µ i − ǫ/2, µ i + ǫ/2]) ≈ αδ(y − µ i )ρ(y, T |x, t), with α > 0 a global constant. In this limit, the optimal control is Regardless the precise value λ −1 θ ≫ 0, the control displays a symmetry breaking at T −t = 1/σ 2 . For earlier times, it is best to steer towards x = 0 (between the targets) and delay the choice of which target to aim for until later. The reason why this is optimal is that from that position the expected diffusion alone is likely to reach any of the slits without control (although it is not clear yet which target). Only sufficiently late in time should one make a choice and steer towards one of the targets, instead of towards the middle. A more careful analysis without using delta functions shows that the phenomenon of symmetry breaking is also present with finite valued targets. Also in that case, the time of symmetry breaking is independent of θ.
For threats, i.e., if the regions around µ i have positive costs and are to be avoided, a similar but reversed phenomenon occurs. See Figure 3. For T − t = 1, the optimal control is outwards the middle for all values of x, trying for a global escape away from both threats (left figure). For later times, if the agent is somewhere in the middle, an escape passing one of the threats would be too risky, and the agent decides to remain in the middle of the two threats. Again, the point  of symmetry breaking does not depend on θ, but the magnitude of the control does. In particular, agents with λ −1 θ < 0 now have big controls, whereas λ −1 θ > 0 seems hardly to care, as we discussed earlier.
A third case that we consider is a target and an adjacent threat: S 1 = {−0.1 < x < 0}, c 1 = −10 and S 2 = {0 < x < 0.1}, c 2 = 10. We did runs for θ = −1, 0, 1, 3. For each θ, we did 1000 runs. Each run started at time t = 0 and position x = 0. End time is T = 1. For each run we monitored the total cost C (i.e. C is control costs plus end cost). Histograms of the log-probability of the cost are plotted in Figure 4. We see that with larger θ, the mode of the distribution shifts to higher costs. On the other hand, the tails of the distribution at the high cost end are thinner with larger θ. This is what is to be expected: small θ is more greedy, aiming at low cost, however at the expense of some outliers with high costs. A larger θ is more cautious, reducing the probability of costly outliers.

DISCUSSION
We showed that path integral control is applicable to risk sensitive control problems. In particular, we showed that the path integral formalism is applicable whenever it is applicable to risk neutral (θ = 0) problems, provided that the path integral does not diverge. Thus the class of stochastic optimal control problems that can be solved exactly is enlarged.
Furthermore, we believe that the path integral solutions also provided insight in what is the consequence of risk sensitive control, in terms of the emerging optimistic and pessimistic behaviour. For example, in the case of both extreme optimistic or pessimistic attitude, we found apathic behaviour in the sense of zero optimal control. A nonzero θ has its effect on the distribution of the cost. In general, we found that smaller θ leads to more target oriented behaviour, while larger θ leads to more cautious behaviour, aimed to avoid threats and high costs.
From a practical point of view, risk sensitive control might be used to make control more greedy or more robust. With lower θ, we found that the mode of the cost distribution decreases, however at the expense of larger tails of this distribution, i.e., more outliers with high cost. With higher θ, these outliers with high costs are prevented.
The path integral method seems a promising approach, with in particular the advantage of the linearity after the log transformation. Its practical applicability is still a subject of research. One application is an algorithm for efficient reinforcement learning applied to a robot dog [19]. Another recent application is in generating animations [1,5]. Whenever path integrals are applied for optimal control aimed to minimize expected cost, it can be applied for risk sensitive optimal control.
General conditions for existence of nontrivial solutions in risk sensitive control is an active area of research, see e.g. [4]. Providing such conditions for risk sensitive path integral control is beyond the scope of this paper and left for future research. Another possibility of future research is to explore whether the generalization or risk sensitive control is also applicable to the more general framework of KL-controls, which include path integral control as a special case [20].