{'fraction_non_alphanumeric': 0.0681039574719433, 'fraction_numerical': 0.015062020082693443, 'mean_word_length': 3.9046929316338352, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates

We consider a risk-sensitive continuous-time Markov decision process over a finite time duration. Under the conditions that can be satisfied by unbounded transition and cost rates, we show the existence of an optimal policy, and the existence and uniqueness of the solution to the optimality equation out of a class of possibly unbounded functions, to which the Feynman-Kac formula was also justified to hold.


Introduction
Risk-sensitive Markov decision processes (in discrete-time) have been studied intensively since 1970s, with one of the pioneering works being [11], and a recent and updated work being [1], to which the interested reader is referred for more references. Compared to the discrete-time framework, there have been fewer works on risk-sensitive CTMDPs (continuous-time Markov decision processes), also known as controlled Markov pure jump processes. An early work on this topic seems to be [18], which obtained verification theorems and solved in closed-form meaningful examples of problems over a fixed time duration. In the recent years, there have been reviving interests in risk-sensitive CTMDPs, see [5,6,21] for problems with a finite horizon, [23] for problems over an infinite horizon, [5,16,22] for problems with average criteria, and [2] for an optimal stopping problem with a more general utility function than the exponential one.
In greater detail, the CTMDP considered in [5] is with bounded transition and cost rates. In [21], the boundedness on the transition rate was relaxed and replaced by a drift-type condition, but the cost rate was still assumed to be bounded. Both papers followed the same line of reasoning: they showed the existence of a solution to the optimality equation, and then showed that the solution coincides with the value function of the problem by applying the Feynman-Kac formula. In Section 7 of [21], the author mentioned that following his method it was unclear how to relax the boundedness assumption on the cost rate at that time, as a suitable version of the Feynman-Kac formula must be established first. The present paper provides a response to this. In greater detail, the main contributions are the following. We provide conditions that allow unbounded transition and (not necessarily nonnegative) cost rates, under which a suitable version of the Feynman-Kac formula was established, and we show that the value function is the unique solution out of a large enough class of functions (possibly unbounded with unbounded derivatives with respect to time) to the optimality equation. It is important for practical applications to consider models with unbounded transition and cost rates. We illustrate this with an example of controlled M/M/∞ queueing system. Compared with [5,21], which concentrated on Markov policies, we consider a more general class of policies. When the cost rate is nonnegative, a different method was followed in [6], which is not based on the Feynman-Kac formula. If the cost rate is nonnegative, then the conditions on the transition and cost rates in [6] are weaker than in the present paper. Moreover, in that general setup of [6], the value function is generally not the unique solution to the optimality equation. In this sense, the present paper also complements [6].
The rest of the paper is organized as follows. In Section 2 we describe the optimal control problem under consideration. Section 3 contains preliminary results, where we establish a version of the Feynman-Kac formula. The optimality results are proved in Section 4. This paper is finished with a conclusion in Section 5.

Model description
Notation: For a Borel space X endowed with the Borel σ-algebra B(X), we denote by C b (X) the space of all bounded continuous functions on X with the norm u := sup x∈X |u(x)|. Throughout this paper, measurability is understood in the Borel sense.
We consider the CTMDP model M := {S, A, A(·, ·), q, c, g} consisting of the following elements. The state space S is a denumerable set, endowed with the discrete topology. The action space A is a (nonempty) Borel space. The multifunction (t, i) ∈ [0, ∞) × S → A(t, i) ∈ B(A) specifies the set of admissible action spaces given the current time and state, and is assumed to be with a measurable graph K := {(t, i, a) ∈ [0, ∞) × S × A : a ∈ A(t, i)}, containing the graph of some measurable mapping from [0, ∞) × S to A. The transition rate is given by a signed kernel q on S given K, assumed to satisfy q(j|t, i, a) ≥ 0 if j = i with j, i ∈ S, q(S|t, i, a) ≡ 0, and where q(t, i, a) := −q(i|t, i, a) ≥ 0 for all (t, i, a) ∈ K. The running cost rate c is a measurable function on K. We shall consider the problem over a finite time duration. The terminal cost g is a function on S.
We briefly describe the construction of a CTMDP as in [14,15]. Let S ∆ := S {∆} (with some ∆ ∈ S being an isolated point), Ω 0 := (S × (0, ∞)) ∞ be the countable product. The canonical sample space Ω is the union of Ω 0 and all the sequences in the form of (i 0 , θ 1 , i 1 , . . . , θ k , i k , ∞, ∆, ∞, . . .) for some k ≥ 0 (accepting θ 0 := 0). Let F be the Borel σ-algebra on Ω. For each ω ∈ Ω, introduce In what follows, the argument ω is often omitted. Let F t be the internal history of the marked point process {T n , X n }. Let T ∞ := lim k→∞ T k . The controlled process {ξ t } is defined by Here and below, I E stands for the indicator function on any set E, and for notational convenience, we defined that i · 0 = 0 and i · 1 = i for each i ∈ S ∆ .
A (history-dependent) policy π is determined and often identified by a sequence of stochastic kernels {π k , k ≥ 0} such that A policy π is called Markov if, with slight abuse of notations, π(da|ω, t) = π(da|ξ t− , t), which is denoted by π t (da|·), where ξ t− = lim s↑t ξ s . A Markov policy π t (da|·) is called deterministic Markov if there exists a measurable mapping f on [0, ∞) × S such that π t (da|i) is a Dirac measure concentrated at f (t, i). A deterministic Markov policy will be denoted by the underlying measurable mapping f . We denote by Π the set of all policies, by Π r m the set of all Markov policies, and by Π d m the set of all deterministic Markov policies.
For each π ∈ Π, the random measure m π defined by is predictable, see [12]. For each π ∈ Π and i ∈ S, let P π i be the probability on (Ω, F) such that P π i (ξ 0 = i) = 1, and with respect to which, m π (j|ω, t)dt is the dual predictable projection of the random measure n≥1 δ (Tn,Xn) (dt, dx) of the marked point process {T n , X n } on B((0, ∞) × S), see [12,14] or Chapter 4 of [15] for more details. Let E π i be the expectation taken with respect to P π i . For the intuitive description, a CTMDP is a continuous-time Markov pure jump process whose local characteristics (transition intensity and post-jump distributions) are controlled. After the n-th jump, and a history of state and sojourn times h n = (i 0 , θ 1 , . . . , θ n , i n ) is observed with θ n < ∞, the conditional (joint) distribution of the next state and sojourn time is determined by A q(j|t n + t, i n , a)π n (da|h n , t − t n )e − t 0 A q(s+tn,in,a)πn(da|hn,s)ds dt, j = i n , where t n is the observed value of the n-th jump moment. In particular, the next sojourn time has the conditional distribution obeying a nonstationary exponential distribution, and a policy specifies the selection of an action at any time moment based on the observed history.
We consider the following optimal control problem over the finite time duration T > 0: Conditions imposed in the next section guarantee that the above expectation and integral are well defined. For each i ∈ S, let 3) is often said to be with a risk-sensitive criterion, as the exponential utility reflects that the decision maker is increasingly averse to the higher cost, see [11]. This is in contrast with a linear utility, which is called risk-neutral. In discrete-time, risk-sensitive Markov decision processes received increasing interest in the recent years, see [3,4,13,17] for example. These works mainly consider infinite-horizon problems; in the discrete-time setup, problems on finite horizon can be readily solved using backward induction. See also [1], which considered a more general utility function.
The objective of this paper is to provide conditions that can be satisfied by unbounded transition and cost rates, under which, there exists a deterministic Markov optimal policy, and the optimality equation has a unique solution out of a certain class of functions. We present an example in the next section, demonstrating a natural application of CTMDPs to controlled queueing system, where the transition and cost rates are both unbounded and thus not covered by the previous literature.

Preliminaries
In this section, we impose a set of conditions allowing one to consider unbounded transition and cost rates, see Example 3.1 below, and present several preliminary statements, which will serve the proof of Theorem 4.1 below.
The immediate and relevant consequences of Condition 3.1 are collected in the next lemma.
Proof. Parts (a) and (b) are known, see e.g., [8,19,20]. We next verify part (c). By part (a), for P π i -almost all ω ∈ Ω, there are finitely many values taken by in where the first inequality is by the Jensen inequality, the second inequality is by Condition 3.1(c), and the last inequality is by part (b).
Part (a) of the previous lemma asserts that under the imposed conditions therein, the controlled process is nonexplosive under each policy. This fact is used in the proof of Theorem 3.1 below, see the first paragraph therein as well as (3.7).
with the function V as the Condition 3.1.
The role of this condition is seen in the proof of Theorem 3.1, where the Cauchy-Schwarz inequality is used, see (3.4) therein. Conditions 3.1 and 3.2 guarantee the growth of the value function and its derivative to be suitably bounded by the weight functions V and V 1 , and it is out of this class of functions that we show the Feynman-Kac formula applies. The previous works [5,21] only showed that the Feynman-Kac formula is applicable to a class of bounded functions, and so confined themselves to the class of bounded cost rates, which excludes some potentially interesting applications. Let us formulate such an example, which are with unbounded transition and cost rates and satisfy Conditions 3.1 and 3.2. This problem can be formulated as a CTMDP with the following primitives. The state space is The running cost rate is given by c(t, i, a) = C 1 i + a, and the terminal cost is given by g(i) = −C 2 i.
Let us introduce some additional notations, which will be needed in the next statement. In particular, it formalizes what we mean in the Introduction by "a large enough class of functions" to which, the Feynman-Kac formula applies. Let X be a Borel space.
Proof. According to Lemma 3.1(a), we concentrate on Ω ′ on which T ∞ = ∞, and hence (3.2) holds. Since By the Cauchy-Schwarz inequality, where the second to the last inequality is obtained by a similar argument to the one for (3.1), and the last inequality is by Lemma 3.1(b). Now it follows from (3.3) that

Now it follows from (3.4) that
with ∆ψ(ω, t, ξ t ) := ψ(ω, t, ξ t ) − ψ(ω, t−, ξ t− ). (Recall that the function ψ(ω, t, j) is absolutely continuous in t over finite interval, and for each fixed ω ∈ Ω ′ with Ω ′ being defined in the beginning of this proof, ξ t (ω) is piecewise constant in t ∈ [0, T ], and assumes finitely many values over that interval.) By (3.5) and (3.6), we take legitimately the expectation on the both sides of the previous equality, and obtain where the last equality holds because the random measure m π defined by (2.2) is the dual predictable projection of the random measure n≥1 δ (Tn,Xn) (dt, dx) on B((0, ∞) × S) under P π i , see p.131 of [15]. The statement is proved.
The above Feynman-Kac formula in the above theorem was justified in [21], see Theorem 3.1 therein, when π is a Markov policy, and ϕ is assumed to be bounded.
The next statement provides a verification theorem, which was known in [18] when the transition rate is bounded.
Proof. Concentrate on Ω ′ as in the proof of the previous theorem. It holds for almost all t ∈ [0, T ] that Now by applying Theorem 3.1 to the deterministic Markov policy f and an arbitrarily fixed π ∈ Π, we see where the first equality holds because ϕ(T, i) = e g(i) , see (3.8); similarly, replacing f for π in the equalities in the above, Since π was arbitrarily fixed, V(f, i) = ϕ(0, i) = V * (i), as required.
According to the previous statement, (3.8) is called the optimality equation, and the policy f in (3.9) is optimal.
The next statement was basically obtained in Theorem 2.1 in [5], see also [21]. The main objective in this paper is to relax the boundedness requirements in the previous statement.

Optimality result
We impose the following condition, which guarantees the existence of an optimal policy.   [10]. This fact will be used in the proof of the next statement.
Also note that Condition 4.1 is satisfied by Example 3.1.
The main optimality result is the following one. Proof. The statement would follow from Corollary 3.1, once we showed the existence of some ϕ ∈ C 1 V,V 1 ([0, T ] × S) satisfying (3.8). We verify this fact following a similar reasoning as in [7] dealing with a risk-neutral CTMDP problem, which was also adopted in [21], dealing with a model with a bounded cost rate. Namely, we shall obtain the desired solution ϕ as a limit point of an equicontinuous family {ϕ n } of functions, which in turn are obtained from a sequence of CTMDP models with bounded transition and cost rates. The denumerable state space serves to prove the equicontinuity of the family {ϕ n }. The details are as follows.
We consider the resulting sequence of CTMDP models M n := {S, A n (t, i), c n , g n , q n } .
Note that the models {M n } are all with bounded transition and cost rates, and so Proposition 3.1 implies, for each n ≥ 1, the existence of a unique ϕ n in C 1 1,1 ([0, T ] × S) and some f n ∈ Π d m satisfying     (The cost rate and the terminal cost were assumed to be nonnegative in [6], but the results obtained there apply because M (s) n has bounded transition and cost rates, which can be reduced to the nonnegative case after one add to the cost rate and the terminal cost a large enough constant.) Thus, we obtain the bound Next, we show that {ϕ n , n ≥ 1} is an equicontinuous family of functions on [0, T ] × S, as follows. Let Then, from Condition 3.1 and (4.2), we see Hence, {ϕ n , n ≥ 1} is equicontinuous at (s 0 , i 0 ), which, together with the arbitrariness of (s 0 , i 0 ) ∈ [0, T ] × S, yields that {ϕ n , n ≥ 1} is equicontinuous on [0, T ] × S. By Arzela-Ascoli theorem, see, e.g., p.96 of [9], there exist a subsequence {ϕ n k , k ≥ 1} of {ϕ n , n ≥ 1} and a continuous function ϕ on [0, T ] × S such that lim k→∞ ϕ n k (s, i) = ϕ(s, i), and |ϕ(s, i)| ≤ M e T ρ V (i) ∀ (s, i) ∈ [0, T ] × S, (4.4) where the last inequality is by (4.2). Let We next verify that lim k→∞ H n k (t, i) = H(t, i) for each (t, i) ∈ [0, T ] × S, as follows. Let (t, i) ∈ [0, T ] × S be arbitrarily fixed. Since q n k (j|t, i, a) → q(j|t, i, a) for all j ∈ S and a ∈ A(t, i) as k → ∞, Since A(t, i) is compact, by taking subsequences if necessary, we can assume without loss of generality that lim inf k→∞ H n k (t, i) = lim k→∞ H n k (t, i) and for some a ∈ A(t, i), f n k (t, i) → a as k → ∞. By the virtue of Lemma 8.3.7 in [10], we have Therefore, we see that ϕ ∈ C 1 V,V 1 ([0, T ]×S). The required deterministic Markov policy f exists because of the fact mentioned below Condition 4.1, a measurable selection theorem, see Proposition D.5 of [9].
Finally, we verify the uniqueness part. Let ϕ ∈ C 1 V,V 1 ([0, T ] × S) be an arbitrarily fixed solution to (3.8). (The above reasoning shows that there exists at least one.) Let s ∈ [0, T ] be fixed, and consider the s-shifted model M (s) = S, A (s) (t, i), q (s) , c (s) , g , which is defined as for the M for some deterministic Markov policy f (s) . By applying Corollary 3.1 to the s-shifted model M (s) , we see ϕ (s) (0, i) = V (s) (i), and thus ϕ(s, i) = V (s) (i) for each i ∈ S. Since s ∈ [0, T ] was arbitrarily fixed, it follows that ϕ is the unique solution to (3.8) out of ϕ ∈ C 1 V,V 1 ([0, T ] × S). The proof is completed.

Conclusion
In this paper, we considered a risk-sensitive CTMDP problem in a denumerable state space over a finite time duration. Under conditions that can be satisfied by unbounded transition and cost rates, the optimality equation was shown to have a unique solution out of a class of functions, to which Feynman-Kac formula was shown to be applicable. The results obtained in this paper can be viewed as a response to the remark in Section 7 of [21], and complemented the relevant results in [6].