{'fraction_non_alphanumeric': 0.0671672473113626, 'fraction_numerical': 0.011555190098024365, 'mean_word_length': 3.8014545454545456, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 26, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Compactly Restrictable Metric Policy Optimization Problems

We study policy optimization problems for deterministic Markov decision processes (MDPs) with metric state and action spaces, which we refer to as Metric Policy Optimization Problems (MPOPs). Our goal is to establish theoretical results on the well-posedness of MPOPs that can characterize practically relevant continuous control systems. To do so, we define a special class of MPOPs called Compactly Restrictable MPOPs (CR-MPOPs), which are flexible enough to capture the complex behavior of robotic systems but specific enough to admit solutions using dynamic programming methods such as value iteration. We show how to arrive at CR-MPOPs using forward-invariance. We further show that our theoretical results on CR-MPOPs can be used to characterize feedback linearizable control affine systems.

Many frameworks extend the theory of discrete MDPs to general state and action spaces. Most relevant for our work, [1] develops semicontinuous MDPs with Borel measurable policy classes, [9] and [10] develop results for analyticially and universally measurable policy classes, respectively, and [11] refines conditions for the wellposedness of value iteration. For a more complete summary, see [12]. Our work focuses on developing classes of policy optimization problems that can naturally connect theoretical results from reinforcement learning with nonlinear continuous-time control, and can be directly translated to checking certain properties in control systems.
Methods that maintain the forward-invariance of subsets of the state space of interest have been well-studied [13]- [15]. These and related methods have recently found applications in safe reinforcement learning [16]- [19]. Physical systems and robotic platforms have been popular applications for reinforcement learning methods recently, with the majority of methods employing function approximation, discretizations (such as fixed or adaptive gridding or state aggregation), or direct policy search [20]- [22]. While policy search methods generally have no guarantees of global optimality, they have received much attention due to the scalability problems of dynamic programming methods (especially after discretization) and convergence problems of approximate dynamic programming (for a more complete discussion of the relative merits of these methods, see [20, §2.3]). In contrast, our goal is to identify settings that are compatible with value iteration and to use control theoretic tools to guide solution methods so we can expect good performance.

A. Contributions
We develop our results in the following phases: 1) We describe a generic property (Assumption 1) that admits the well-posedness of value iteration for MPOPs (Theorem 1). 2) We leverage the forwardinvariance of compact sets to design MPOPs and policy classes that satisfy Assumption 1, allowing us to prove well-posedness of a large class of policy optimization problems (Theorem 2). Such problem settings are called CR-MPOPs. 3) We apply our results on CR-MPOPs to analyze control affine systems with time-sampled control inputs. 4) We further apply our results to analyze robotic systems with time-sampled control inputs, which comprise a large class of control affine systems. These results translate the generic requirements for well-posedness to concrete requirements on the robotic system. 5) We finally show that full-state feedback linearizable control affine systems can render a compact subset of the state space forward invariant with continuous controllers (thus satisfying conditions of Theorem 2), demonstrating practically relevant instances of CR-MPOPs.

B. Notation, Conventions, and Definitions
We consider nonempty metric spaces throughout this paper and endow each such space with the σ-algebra generated by its topology, called the Borel σ-algebra. In this case, open sets and closed sets are measurable sets and continuous functions are measurable functions. A metric space is separable if it contains a countable dense subset. Finite or countable sets can be regarded as separable metric spaces when equipped with the discrete metric [23, §3.A]. For nonempty metric spaces X and Y , the set of measurable functions from X to Y is denoted L 0 (X; Y ). A bounded measurable function f : X → R is upper semicontinuous if for any c ∈ R, the preimage f −1 ([c, ∞)) = {x ∈ X : f (x) ≥ c} is closed.
The set of all subsets of Y is the powerset of Y , denoted P(Y ). A set-valued function C : X → P(Y ) is called a correspondence. The graph of C is defined as: If C(x) = ∅ for all x ∈ X, then a selector of C is a function f : X → Y satisfying f (x) ∈ C(x) for all x ∈ X. For any set B ⊆ Y , the lower preimage of B under C is defined as: The correspondence C is upper hemicontinuous 1 if the lower preimage of each closed set is a closed set [24,Lemma 17.4].
A nonempty subset X 0 ⊆ X is a metric space when equipped with the restriction of the metric on X to X 0 × X 0 . For any open set U ⊆ X 0 , there is an open set V ⊆ X such that U = X 0 ∩V . For any continuous function f : X → Y , the restriction f | X 0 : X 0 → Y is continuous. Similarly, for any measurable set A ⊆ X 0 , there is a measurable set B ⊆ X such that A = X 0 ∩ B, and for any measurable function f : X → Y , the restriction f | X 0 is measurable.
Consider a nonempty metric space Z and a function f : X × Y → Z. For x ∈ X and y ∈ Y , the x-section fx : Y → Z and the y-section f y : X → Z are defined as: for all x ∈ X and y ∈ Y . If f is continuous, then f has continuous sections, and if f is measurable, then f has measurable sections. We also define sections for functions on graphs. Consider a correspondence C : X → P(Y ) with C(x) = ∅ for all x ∈ X, and a function f : Graph(C) → Z. For x ∈ X, we define the x-section fx : C(x) → Z as above. N denotes the natural numbers, Z + denotes the nonnegative integers, R + and R ++ denote nonnegative and positive reals, and S d + and S d ++ denote d×d positive semidefinite and definite matrices.

II. METRIC POLICY OPTIMIZATION PROBLEMS
In this paper, a deterministic Markov decision process (MDP) will be characterized by a tuple (S, A, C, f, r, γ), with state space S, action space A, action-admissibility correspondence C : S → P(A) satisfying C(s) = ∅ for each state s ∈ S, transition map f : Graph(C) → S, reward function r : Graph(C) → R, and discount factor γ ∈ [0, 1). Note while r may be unbounded, r cannot assume the values ±∞. We refer to an MDP as a metric MDP if S and A are nonempty separable metric spaces and f and r are measurable functions. In this paper, we focus solely on metric MDPs, and often refer to them simply as MDPs for brevity.
We will limit our consideration to deterministic, Markovian, and stationary (time-invariant) policies. In this case, a policy π : S → A is a selector of C; that is, π(s) ∈ C(s) for all states s ∈ S. The corresponding closed-loop transition map fπ : S → S and singlestep reward function rπ : S → R are defined as: fπ(s) = f (s, π(s)), rπ(s) = r(s, π(s)).
for all states s ∈ S. For any t ∈ Z + , let f t π : S → S denote the t-iterated composition of fπ. Define the subset Sπ ⊆ S as: γ t rπ(f t π (s)) converges absolutely . (5) 1 Some authors call such correspondences upper semicontinuous; we use hemicontinuous to distinguish correspondences from real-valued functions.
A set of policies Π admits a partial order if Sπ = S for all policies π ∈ Π. In this case, for policies π, π ∈ Π, if Vπ(s) ≥ V π (s) for all states s ∈ S, then π π .
Definition 1 (Metric Policy Optimization Problem). We refer to the pair of an MDP (S, A, C, f, r, γ) and a policy class Π admitting a partial order as a metric policy optimization problem (MPOP). We call an MPOP well-posed if there is an optimal policy π * ∈ Π with π * π for all policies π ∈ Π.
Goals of this section. Our goal is to analyze well-posedness of policy optimization for MDPs. We first describe sufficient conditions for well-posedness of value iteration (Theorem 1). We then establish conditions under which ill-posed MPOPs can be restricted to wellposed problems by only considering policies that render the same subset of the state space forward-invariant (Theorem 2), resulting in CR-MPOPs. We will show how to apply our results on value iteration to control affine systems in Section III.
Throughout this section, we refer to and modify the following example to ground our presentation: Example 1. Consider the state space S = R, action space A = R, and the constant action-admissibility correspondence C = A. Additionally, consider the transition function f : Graph(C) → S and reward function r : Graph(C) → R defined as: for all state-action pairs (s, a) ∈ Graph(C), and a discount factor γ ∈ [0, 1). Note that |f (s, a)| ≤ |s| + | tanh (a)| ≤ |s| + 1 for all state-action pairs (s, a) ∈ Graph(C). Consider any policy π : S → A. By induction, we have |f t π (s)| ≤ |s| + t for all states s ∈ S and t ∈ Z + . For any state s ∈ S and T ∈ Z + , we have: The sequence of partial sums as T → ∞ is monotone and bounded below, so Vπ(s) is well-defined. As s was arbitrary, we have Sπ = S.

A. Value Iteration
Consider a policy class Π with Sπ = S for each policy π ∈ Π and sup π∈Π Vπ(s) < ∞ for each state s ∈ S. Accordingly, we define the optimal (with respect to Π) value function V * : S → R as: for all states s ∈ S. If a policy π * ∈ Π satisfies V π * = V * , then π * is optimal and the policy optimization problem is well-posed. Value iteration generates a sequence of approximations of V * . Given an initial guess V 0 : S → R, we seek a sequence of realvalued functions {Vn : n ∈ N} satisfying: for all states s ∈ S and n ∈ Z + . We now describe conditions under which MPOPs are well-posed and can be solved with value iteration (Theorem 1). When assumptions are strengthened by requiring S and A to be Polish spaces, we will make use of [11,Theorem 4.1] (the setting of this theorem is called a semicontinuous model in [1, §6.7]). We provide a more direct proof of well-posedness without these additional assumptions in the appendix. Assumption 1. The action admissibility correspondence C has compact values and is upper hemicontinuous, the transition function f is continuous, the reward function r is upper semicontinuous and bounded, and the policy class Π is the set of all measurable selectors of C. Moreover, for each state-action pair (s, a) ∈ Graph(C), there is a corresponding policy πs,a ∈ Π satisfying πs,a(s) = a. Remark 1. If the projection of Graph(C) onto the action space is compact, then the assumption that for any state-action pair (s, a) ∈ Graph(C), there exists a policy πs,a ∈ Π with πs,a(s) = a can be removed. See Lemma 1.
for all state-action pairs (s, a) ∈ Graph(C ). Note that Graph(C ) = S×[−1, 1], so r is bounded. Since f is continuous, its restriction to Graph(C ) is as well. With Π the set of all measurable selectors of C , note that for every pair (s, a) ∈ Graph(C ), the constant policy πs,a = a is a measurable selector of C , so πs,a ∈ Π. Thus, the MPOP defined by the MDP (S, A, C , f | Graph(C ) , r , γ) and policy class Π satisfies Assumption 1.
We now demonstrate well-posedness under Assumption 1.
Theorem 1. Consider an MPOP characterized by an MDP (S, A, C, f, r, γ) and a policy class Π that satisfies Assumption 1.
There is an optimal policy π * ∈ Π satisfying V π * = V * , and V * is the limit of value iteration when the initial guess is bounded and upper semicontinuous.
Proof. If S and A are Polish spaces, we can show that the MDP in question satisfies the three conditions of Assumptions (W*) of [11], in which case, there is an optimal policy and V * is the limit of value iteration when the initial condition is the zero function (a conclusion of [11,Theorem 4.1]). The first condition follows since r is upper semicontinuous and bounded above (as it is bounded both above and below), which corresponds to a lower semicontinuous cost function that is bounded below. The second condition is satisfied by the upper hemicontinuity of C (using the sequential definition in [24,Theorem 17.20]) since r is also bounded below. The third condition is satisfied since the MDP is deterministic (for which transition probabilities are represented by Dirac measures). In this case, we require: lim n→∞ ϕ(f (sn, an)) = ϕ(f (s, a)), for any bounded continuous function ϕ : S → R and any convergent sequence {(sn, an) ∈ Graph(C) : n ∈ N} converging to (s, a) ∈ Graph(C); equality follows since ϕ • f is continuous (as the composition of continuous functions). If S and A are not both Polish spaces, we prove well-posedness in the appendix, in which case, V * is the limit of value iteration when the initial guess is any bounded and upper semicontinuous function.
We conclude with a lemma mentioned in Remark 1, allowing Assumption 1 to be relaxed slightly. Lemma 1. Suppose an MDP (S, A, C, f, r, γ) and policy class Π satisfy Assumption 1, suppose the projection of Graph(C) onto A is compact, and let ρ A : A × A → R + denote the metric on A. For any state-action pair (s, a) ∈ Graph(C), the function ϕs,a : Graph(C) → R defined as: for all state-action pairs (s , a ) ∈ Graph(C) admits a maximizing policy πs,a ∈ Π for which πs,a(s) = a.
Proof. Since ρ A is continuous, so is ϕs,a. Since the projection of Graph(C) onto A is compact, ϕs,a is bounded. If S and A are Polish spaces, then the proof of [11,Lemma 3.4] uses the Arsenin-Kunugui theorem to show the existence of a policy πs,a ∈ Π satisfying: ϕs,a(s , πs,a(s )) = max for all states s ∈ S. In particular, πs,a(s) = a as ϕs,a(s, πs,a(s)) = − min a ∈C(s) ρ A (a , a) = 0. This requires Assumptions (W*) of [11] to be met, which is verified in the proof of Theorem 1. We arrive at the same conclusion when S and A are not both Polish spaces in the appendix.

B. Compactly Restrictable MPOPs
While Assumption 1 allows us to determine when MPOPs are well-posed and can be solved using value iteration, it is not met for many systems of interest. We now outline an alternative assumption that not only mitigates theoretical shortcomings, but also better fits the problem settings for physical systems. Our main result here is Theorem 2, showing how an MPOP satisfying these new assumptions can generate an MPOP satisfying Assumption 1 through a systematic transformation. We call the original MPOP a CR-MPOP. 1) Motivation: Consider an MPOP defined by an MDP (S, A, C, f, r, γ) and a policy class Π that satisfies Assumption 1. The proof of Theorem 1 requires a bounded reward function r and a compact-valued action-admissibility correspondence C. For many examples, these requirements are too strict, as shown in Example 2. If C can be modified such that its graph is compact and if r is continuous, then r will be bounded over the graph of the modified correspondence. This guides our systematic construction of a wellposed policy optimization problem from an ill-posed one.
If the graph of the modified correspondence is compact, the projection of the graph onto the state space must also be compact. Therefore, if the state space is not already compact, then the modified correspondence must be defined on a compact strict subset S 0 ⊂ S; that is, the modified correspondence must have the form C 0 : S 0 → P(A), where C 0 (s) ⊆ C(s) is nonempty and compact for all states s ∈ S 0 . Moreover, while restricting admissible actions to compact sets is reasonable for physical systems (as forces, voltages, currents, etc. are bounded by physical constraints), these restrictions must be chosen carefully to enforce the nonemptiness condition. For the restriction of f to Graph(C 0 ) to be a well-defined transition function, we require f (s, a) ∈ S 0 for all pairs (s, a) ∈ Graph(C 0 ). Finally, we must ensure that C 0 admits measurable selectors.
Restricting our attention to a compact subset S 0 is also reasonable for many physical systems, as dynamics models are often wellunderstood only in a bounded set around an operating condition.
2) Compact Restriction: For this construction, we use policies that render compact subsets of the state space forward-invariant. A policy π ∈ Π renders a subset S 0 ⊆ S forward-invariant if fπ(S 0 ) ⊆ S 0 . By induction, we have f t π (S 0 ) ⊆ S 0 for any t ∈ Z + . Definition 2 (Compactly Restrictable Metric Policy Optimization Problem). An MPOP characterized by an MDP (S, A, C, f, r, γ) and a policy class Π is a compactly restrictable metric policy optimization problem (CR-MPOP) if there is a nonempty compact subset S 0 ⊆ S and a continuous policy π 0 ∈ Π rendering S 0 forward-invariant.
Assumption 2. The action-admissibility correspondence C has closed values and Graph(C) is closed, the transition function f and reward function r are continuous, and the policy class Π is the set of all measurable selectors of C.
To show how an CR-MPOP satisfying Assumption 2 can lead to an MPOP satisfying Assumption 1, we need the following lemma.

Lemma 2.
Consider an MDP (S, A, C, f, r, γ) and a policy class Π satisfying Assumption 2. For any state s ∈ S, the s-section fs : C(s) → S is continuous. Additionally, for any state s ∈ S and any closed set G ⊆ S, the preimage (fs) −1 (G) is a closed subset of A.
Proof. Let id : S ×A → S ×A denote the identity function on S ×A. For any state s ∈ S, the s-section ids : We now present our guarantee of the existence of a well-posed value iteration settings under Assumption 2.
Theorem 2. Consider a CR-MPOP characterized by an MDP (S, A, C, f, r, γ) and a policy class Π that satisfies Assumption 2. There exists an MPOP characterized by an MDP (S 0 , A 0 , C 0 , f 0 , r 0 , γ) and a policy class Π 0 satisfying Assumption 1, with S 0 ⊆ S rendered forward-invariant by a continuous policy in Π, , and: Proof. For some n ∈ N, consider continuous policies π 1 , . . . , πn ∈ Π satisfying fπ i (S 0 ) ⊆ S 0 for all i ∈ {1, . . . , n}. Such policies are guaranteed to exist by the existence of the continuous policy π 0 ∈ Π. Consider the union of the images of S 0 under each of these policies, defining the compact set A 0 = n i=1 π i (S 0 ). Since S and A are separable metric spaces, so are S 0 and A 0 [24, Corollary 3.5].
Define the correspondence C 0 : S 0 → P(A 0 ) as: for all states s ∈ S 0 . Since π 1 (s), . . . , πn(s) ∈ C 0 (s), C 0 (s) = ∅ for all s ∈ S 0 . By Lemma 2 and since S 0 is closed (it is compact in a metric space), the preimage (fs) −1 (S 0 ) is a closed subset of A. C 0 (s) is thus compact, as it is a closed subset of A 0 . For any closed set G ⊆ A 0 , the lower preimage of G under C 0 satisfies: where p S : S × A → S denotes the canonical projection onto S.
Since G is a closed subset of the compact set A 0 , G is compact, and since S 0 is compact, the product S 0 × G is compact. As a closed subset of a compact set, is a closed subset of S 0 , and since G was arbitrary, C 0 is upper hemicontinuous. As restrictions of continuous functions, f 0 and r 0 are continuous, implying r 0 is upper semicontinuous. Note that: Since A 0 is trivially a closed subset of A 0 , we have already shown The policy class Π 0 satisfies: To verify the last equality, first consider a policy π ∈ Π satisfying π(s) ∈ C 0 (s) for all states s ∈ S 0 . The restriction π| S 0 is a measurable function from S 0 to A 0 selecting from C 0 . Conversely, consider a measurable function π : S 0 → A 0 selecting from C 0 . Pick any policy πe ∈ Π and define the extensionπ : S → A as π on S 0 and πe on S \ S 0 . For any measurable set B ⊆ A, we have: which is a measurable set; therefore,π is a measurable function. Note thatπ(s) = π(s) ∈ C 0 (s) ⊆ C(s) for all states s ∈ S 0 and π(s) = πe(s) ∈ C(s) for all states s / ∈ S 0 . Sinceπ is a measurable selector of C, we haveπ ∈ Π, and inclusion follows sinceπ| S 0 = π.
Finally, since Graph(C 0 ) is compact, its projection onto A 0 is compact. Therefore, by Lemma 1, for every state-action pair (s, a) ∈ Graph(C 0 ), there is a policy πs,a ∈ Π 0 satisfying πs,a(s) = a.
Remark 3. An MPOP characterized by an MDP (S, A, C, f, r, γ) and a policy class Π that satisfies Assumption 1 is trivially a CR-MPOP satisfying Assumption 2 if the following sufficient conditions are met: 1) The state space S is compact, 2) The reward function r is continuous, 3) The policy class Π contains a continuous policy.
If f , r, and π 0 are continuous, then so are fπ 0 and rπ 0 which we can show renders Vπ 0 continuous. This implies r+γVπ 0 •f is continuous. If r is bounded, then so are Vπ 0 and r +γVπ 0 •f . Since r +γVπ 0 •f is upper semicontinuous and bounded, we can show (using [11] or the appendix) that for any state s ∈ S, the optimization problem: is solved by the next policy π 1 , a measurable selector of C. However, we cannot conclude that r + γVπ 1 • f is upper semicontinuous and bounded, so the same argument cannot be applied iteratively.
Demonstrating that policy iteration can be applied requires special knowledge that the policy improvement step in (23) continues to produce policies with value functions that permit maximization and appropriate selection (for the chosen policy class). Such knowledge is available in linear-quadratic regulator (LQR) problems with linear policy classes (see [25] for the discounted case).

D. Computation
There are 3 central difficulties in computationally implementing value iteration for a well-posed MPOP: 1) Representing iterates during value iteration, 2) Determining admissible actions, and 3) Approximating the update step (10). Typically, iterates are represented using function approximators such as neural networks [5] or Gaussian process regression models [26]. Training such approximators involves sampling sufficiently many states from the state space region of interest. For any given state, the maximization in the update step (10) is generally a nonconvex optimization, with possible approximations including (projected) gradient ascent from several initial action seeds or maximization over a finite but sufficiently dense sampling of admissible actions. If the function approximation and optimization approximation errors can be controlled, approximate convergence of value iteration can be guaranteed [27,Proposition 2.3.2].
Any approximate optimization approach requires checking admissibility of actions, which, in the case of a CR-MPOP, requires checking whether or not a state-action pair is mapped to the correct compact set under the transition map. This necessitates efficient set membership checking, and for nonconvex compact sets, proximitybased membership approximations may be needed.
Finally, in Section III-C, we will use the closure of a reachable set (under a specific policy) to restrict problems with feedback linearizable control affine systems to well-posed MPOPs. To sample a state from the reachable set, we can sample a state from the appropriate set of initial conditions and follow the policy for a number of steps sampled from a geometric distribution (with success probability 1 − γ). The resulting distribution is the discounted state distribution under the policy [28], which is supported on the entirety of the reachable set (thus, on a dense subset of its closure).
There are many possible combinations of computational approaches that may be suitable for well-posed MPOPs; we deem the precise best choice outside the scope of this theoretical framing.

III. CONTROL AFFINE SYSTEMS
We now show how to apply the results from Section II to a general class of control systems called control affine systems [29, §13.2.3]. For state and action space dimensions d, m ∈ N, respectively, consider a set D ⊆ R d and vector fields f 0 , g 1 , . . . , gm : D → R d . Define the matrix-valued function G : D → R d×m with columns g 1 , . . . , gm. Define F : D × R m → R d as: for all x ∈ D and u = (u 1 , . . . , um) ∈ R m . An initial value problem with constant control input u ∈ R m is characterized by an initial condition x ∈ D and an open time interval I ⊆ R with 0 ∈ I; a corresponding solution is a differentiable function φ : I → D satisfying φ(0) = x andφ(t) = d dt φ(t) = F (φ(t), u) = f 0 (φ(t)) + G(φ(t))u for all times t ∈ I.

A. Time Sampling
In contrast to continuous-time control design, we consider sampled-data control design (see [6]- [8]), in which initial value problems with constant control characterize the evolution of a system over fixed sample intervals, resulting in control input trajectories that are piecewise constant in time. Such assumptions are realistic for physical systems interacting with digital controllers, which measure states and compute control inputs at nearly fixed frequencies. This setting requires the time intervals over which solutions are defined to be sufficiently long, uniformly for all initial conditions and control inputs under consideration.
Specifically, fix a sample period h ∈ R ++ , and define the subset S h ⊆ D and correspondence C h : S h → P(R m ) such that the following properties are satisfied: • For every initial condition x ∈ S h , the corresponding set of control inputs C h (x) ⊆ R m is nonempty, Define f h : Graph(C h ) → D for all pairs (x, u) ∈ Graph(C h ) as: Example 4 (Linear System). Let D = R d . For matrices A ∈ R d×d and B ∈ R d×m , suppose f 0 (x) = Ax and G(x) = B for all initial conditions x ∈ R d . Then S h = R d and C h (x) = R m for all x ∈ R d . Moreover, f h can be expressed explicitly as a linear function in terms of the matrix exponential of A.

Example 5 (Continuously Differentiable and Lipschitz Continuous
Vector Fields). Suppose D = R d and f 0 , g 1 , . . . , gm are continuously differentiable and (globally) Lipschitz continuous. For a control input u ∈ R m , since the u-section F u : R d → R d is a linear combination of continuously differentiable and Lipschitz continuous vector fields, it is continuously differentiable and Lipschitz continuous. Therefore, for any initial condition x ∈ R d the initial value problem characterized by initial condition x, control input u, and time interval R has a unique solution [30, Theorem 3.1.3]. Therefore, In general, f h does not have an closed-form representation.

Assumption 3.
The vector fields f 0 , g 1 , . . . , gm are locally Lipschitz continuous. Moreover, there exist subsets S ⊆ S h and A ⊆ R m and a correspondence C : S → P(A) such that: • For every initial condition x ∈ S, the set of control inputs

Both Examples 4 and 5 satisfy Assumption 3 with
The assumption of local Lipschitz continuity in Assumption 3 ensures that the restriction of f h to Graph(C) is continuous; this follows from [31, Theorem 3.5] by treating control inputs as parameters.

B. Robotic Systems
Many robotic systems can be modeled as control affine systems, and we will demonstrate how to show that such systems can satisfy Assumption 2, establishing well-posedness of value iteration for a large class of practically relevant MPOPs. Examples of such systems include manipulators, automobiles, aircraft, and spacecraft [32], [33].
1) Dynamics: An unconstrained robotic system with n ∈ N degrees of freedom is characterized by an n-dimensional C 2 manifold Q called the configuration manifold. In this work, we consider open subsets Q ⊆ R n for which we can identify the tangent bundle T Q with Q × R n . The inertia matrix function D : Q → S n ++ characterizes the kinetic energy function T : Q × R n → R ++ , defined as T (q,q) = 1 2q D(q)q for all configurations q ∈ Q and velocitiesq ∈ R n . The assumption that D takes positive definite values ensures that no configurations exist that admit arbitrarily large velocities without affecting the kinetic energy of the system. We require D to be differentiable, allowing us to express the matrixvalued function C : Q × R n → R n×n of Coriolis and centrifugal terms 2 as: for all configurations q = (q 1 , . . . , qn) ∈ Q, velocitiesq = (q 1 , . . . ,qn) ∈ R n , and indices i, j ∈ {1, . . . , n} [32,Equation 4.21]. We also require the potential energy function U : Q → R n to be differentiable, and we denote external nonconservative forces and torques applied to the system with the vector-valued function F ext : Q × R n → R n . Finally, if the system is controlled with m ∈ N actuators, then B : Q → R n×m denotes the actuation matrix function, converting control inputs to forces and torques. With d = 2n and D = Q × R n , we define F : D × R m → R d as: for all x = (q,q) ∈ Q × R n and u ∈ R m . 2) Regularity: We now establish regularity conditions that enable us to define a valid transition function that can be used in an MDP. We first establish sufficient conditions such that every initial condition and control input under consideration correspond to initial value problems with unique solutions. We then establish sufficient conditions for these solutions to exist for all nonnegative time.
Assumption 4. The configuration manifold satisfies Q = R n . The functions D, C, ∇U , F ext , and B are each locally Lipschitz continuous. There is a strictly positive lower bound λ min such that λ min In D(q) for all configurations q ∈ R n . There is some closed set G ⊆ R m such that for any control input u ∈ G there are corresponding constants c 0 ∈ R + and c 1 ∈ R ++ such that: for all configurations q ∈ R n and velocitiesq ∈ R n .

Remark 4.
If the derivative of D is locally Lipschitz continuous, then so is D itself as it is continuously differentiable, and so is C as it is bilinear in the derivative of D and the velocities. If D is twice continuously differentiable, then the derivative of D is locally Lipschitz continuous, implying D and C are as well. If U is twice continuously differentiable, then ∇U is locally Lipschitz continuous.
Remark 5. If B is bounded by some M ∈ R + and there are constants d 0 ∈ R + and d 1 ∈ R ++ such that: for all configurations q ∈ R n and velocitiesq ∈ R n , then for any u ∈ R m , we have: for all configurations q ∈ R n and velocitiesq ∈ R n . Therefore, choose c 0 = d 0 + M u 2 and c 1 = d 1 , as well as G = R m .
The proof of [34,Theorem 9.8b] shows that matrix inversion is locally Lipschitz continuous. Fixing a control input u ∈ G, the u-section F u is locally Lipschitz continuous as it is comprised of sums and products of locally Lipschitz continuous functions. Fix an initial configuration q 0 ∈ R n and velocityq 0 ∈ R n . There is a corresponding maximal open interval Imax ⊆ R with 0 ∈ Imax such that the initial value problem characterized by initial condition (q 0 ,q 0 ), control input u, and time interval Imax has a unique solution φ : Imax → R 2n [30, Theorem 2.4.1] (the proof of this theorem applies to locally Lipschitz continuous vector fields, not just continuously differentiable vector fields). Let ψ : Imax → R n satisfy φ(t) = (ψ(t),ψ(t)) for all times t ∈ Imax, and note that ψ(0) = q 0 andψ(0) =q 0 . Let I + max = R + ∩Imax. The kinetic energy satisfies: T (ψ(t),ψ(t)) − T (q 0 ,q 0 ) = t 0ψ (s) (F (ψ(s),ψ(s)) − ∇U (ψ(s)) + B(ψ(s))u) ds for all times t ∈ I + max . Since T (q,q) ≥ 1 2 λ min q 2 2 for all configurations q ∈ R n and velocitiesq ∈ R n , we have: for all times t ∈ I + max . For any upper bound t f ∈ I + max with t f > 0, the functions φ 1 , φ 2 : [0, t f ] → R + defined as: for all times t ∈ [0, t f ]. By the Gronwall-Bellman inequality [31, Lemma A.1], we have: for all times t ∈ [0, t f ]. Since t f was arbitrary, the bound (36) holds for all times t ∈ I + max . We now use (36) to show that I + max = R + . For contradiction, assume there is some time t f ∈ R ++ with t f / ∈ I + max . By upper bounding the right-hand side of (36) by its value at the time t f , we conclude thatψ is bounded on I + max . Also: for all times t ∈ I + max ; that is, ψ is bounded on I + max . Therefore, φ is bounded on I + max , contradicting [30, Theorem 2.4.3] (again, the proof of this theorem applies to locally Lipschitz continuous vector fields). This implies I + max = R + , meaning for any sample period h ∈ R ++ , there is an open interval I ⊆ R with [0, h] ⊂ I.
Thus, any robotic system satisfying Assumption 4 also satisfies Assumption 3, no matter which sample period h is chosen.

C. Sampled-Data Control
We now determine sufficient conditions for the existence of a continuous policy rendering a compact subset of the state space forward-invariant. This construction uses the methods of [7], [8], based on the original work [6] in sampled-data control. A control affine system is full-state feedback linearizable if D is open and there exist a function Φ : D → R d that is a diffeomorphism between D and an open subset of R d , a continuous controller k fbl : R d × R m → R m that accepts an auxiliary input, and a controllable pair (A, B) ∈ R d×d × R d×m such that: for all x ∈ D and auxiliary control inputs v ∈ R m . Remark 6. Any robotic system satisfying Assumption 4 with n = m and B(q) invertible for all configurations q ∈ R n is full-state feedback linearizable. The feedback linearizing controller k fbl : R d × R n → R n is defined as: for all x = (q,q) ∈ R n × R n and v ∈ R n , and: for all x = (q,q) ∈ R n × R n and v ∈ R n .
Consider a full-state feedback linearizable control affine system, and suppose 0 d ∈ D and Φ(0 d ) = 0 d . Consider any gain matrix K ∈ R m×d making A − BK Hurwitz stable with all eigenvalues having strictly negative real parts. Suppose the system satisfies Assumption 3 and 0 d ∈ int(S). Additionally, suppose π 0 : S → R m defined as: for all states x ∈ S satisfies π 0 (x) ∈ C(x) for all states x ∈ S.
Since k fbl and Φ are continuous, π 0 is continuous. By [7,Lemmas 3,4], there is a continuous (class KL) function β : R + × R + → R + with monotonically increasing s-sections β s for all s ∈ R + and monotonically nonincreasing r-sections βr for all r ∈ R + , as well as some R ∈ R ++ and a bounded open set N ⊂ R d with 0 d ∈ N and: Additionally, for any R ≤ R, if the sample period is sufficiently small, we have: for all x ∈ N and t ∈ Z + . Let R ⊆ S denote the set of all states reachable from N when following π 0 , defined as R = {(f h ) t π 0 (x) ∈ S : x ∈ N, t ∈ Z + }, and note that (f h )π 0 (R) ⊆ R and cl(R) ⊆ S. Since (f h )π 0 (cl(R)) ⊆ cl(R) ⊆ S, we can set S 0 = cl(R). The set S 0 ⊆ S is closed and bounded since: Therefore, S 0 is a compact set rendered forward-invariant by the continuous policy π 0 , implying the system satisfies Assumption 2. Finally, we can construct a compact subset of the state space in a similar manner by appealing to [8,Theorem 3], employing the notion of practical safety. In this case, if a compact 0-superlevel set of a family of sampled-data control barrier functions [8,Definition 7] can be rendered forward-invariant when the transition map is approximated by an appropriate Euler/Runge-Kutta scheme, then an enlarged (but bounded) set can be rendered forward-invariant for the exact transition map when the sample period is sufficiently small. The enlargement can be controlled by decreasing the sample period.

IV. FUTURE WORK
There are many directions for future work. First, our results can be viewed as "existence" results, and set the stage for studying computational aspects of policy optimization in metric MDPs. Second, our results are applicable to infinite-dimensional state and action spaces, such as those encountered in the control of systems governed by partial differential equations. This setting provides new challenges, such as how to choose appropriate compact subsets of the state space, as well as further computational considerations. Third, and it would be interesting to also consider stochastic dynamics. Finally, it is important to study the interaction between policy optimization and model learning when a model must be derived from data.

APPENDIX
We offer alternative, direct proofs of Theorem 1 and Lemma 1. We proceed as follows: 1) Introduce additional notation, conventions, and definitions, 2) Show the measurability and boundedness of value functions under measurable policies, 3) Introduce Bellman and optimal Bellman operators, 4) Prove a selection theorem for bounded and upper semicontinuous functions on graphs of correspondences, 5) Connect these results to proofs of Theorem 1 and Lemma 1.

A. Additional Notation, Conventions, and Definitions
For a nonempty metric spaces X and Y , the set of bounded measurable functions from X to R is denoted L 0 b (X), which is a Banach space when equipped with the norm · sup : L 0 b (X) → R, defined as: for all bounded measurable functions f : X → R. The set of bounded upper semicontinuous functions from X to R is denoted C u b (X), the set of bounded continuous functions from X to R is denoted C 0 b (X), and note that . For another nonempty metric space Y , consider a correspondence C : X → P(Y ). C is measurable if the lower preimage of each closed set is a measurable set, and C is weakly measurable if the lower preimage of each open set is a measurable set. If C is measurable, then it is weakly measurable [24,Lemma 18.2].
The product X × Y is a metric space, with a topology generated by the basis {U × V : U ⊆ X and V ⊆ Y are open}, and, if X and Y are separable, a σ-algebra generated by the collection For another nonempty metric space Z, consider a function f : X × Y → Z. If fx is continuous for each x ∈ X and f y is measurable for each y ∈ Y , then f is called a Carathéodory function; if Y is also separable, then f is measurable [24,Lemma 4.51].
In what follows, consider an MPOP characterized by an MDP (S, A, C, f, r, γ) and a policy class Π satisfying Assumption 1.

B. Measurability and Boundedness
To establish the measurability of the value function under a policy π ∈ Π, we begin by establishing measurability of the corresponding closed-loop transition map fπ and single-step reward function rπ. To this end, we define z : S → S × A as: for all states s ∈ S. Consider measurable sets A ⊆ S and B ⊆ A; the preimage of the product A × B under z is z −1 (A × B) = {s ∈ S : s ∈ A, π(s) ∈ B} = A ∩ π −1 (B), which is a measurable set since π is a measurable function. Since measurable products generate the σ-algebra on S × A, z is measurable, implying the compositions fπ = f • z and rπ = r • z are measurable functions. It follows that for any t ∈ Z + , the t-iterated composition f t π is a measurable function, as is the composition rπ • f t π . Note that Sπ = S since rπ is bounded (by the same bound on r); therefore, the corresponding value function Vπ : S → R is well-defined on all of S. Moreover, Vπ is measurable since it is the pointwise limit of a sequence of partial sums of measurable functions. Finally, Vπ is bounded by (1 − γ) −1 times the bound on r, so Vπ ∈ L 0 b (S).

C. Bellman Operators
The Bellman operator Tπ : L 0 b (S) → L 0 b (S) under policy π generalizes the implicit structure of the value function Vπ, satisfying: [Tπ(g)](s) = rπ(s) + γg(fπ(s)), for all states s ∈ S, for all bounded measurable functions g : S → R. Indeed, for any bounded measurable function g : S → R, the function Tπ(g) = rπ + γg • fπ is bounded and measurable. The Bellman operator Tπ is a γ-contraction [3, §1.4.1], so by the contraction mapping principle [24,Theorem 3.48], Vπ is the only bounded measurable function with Tπ(Vπ) = Vπ. We use Bellman operators to characterize the optimal Bellman operator, which is commonly used in value iteration. Since each value function corresponding to a policy in Π is bounded by (1 − γ) −1 times the bound on r, the optimal value function V * is well-defined and bounded by the same bound. The optimal Bellman operator T : for all states s ∈ S is well-defined, upper semicontinuous, and bounded. Moreover, there is a policy πϕ ∈ Π satisfying: for all states s ∈ S.
Proof. The function gϕ is well-defined and upper semicontinuous by [24,Lemma 17.30] and is bounded by the bound on ϕ.
Define the correspondence Cϕ : S → P(A) as: where p A : S × A → A is the continuous function projecting S × A onto A, the set of maximizing actions Cϕ(s) is compact. The remainder of the proof is a modification of the proof of [1, Theorem 2.5.A]. Since C is upper hemicontinuous, it is measurable. Therefore, C is weakly measurable [24,Lemma 18.2]. Let ρ A : A × A → R + denote the metric on A, and define the distance function ρ C : S × A → R + as: for all states s ∈ S and actions a ∈ A. Since C is weakly measurable, ρ C is a Carathéodory function [24,Theorem 18.5]; that is, for every state s ∈ S, the s-section (ρ C )s is continuous, and for every action a ∈ A, the a-section (ρ C ) a is a measurable function.
Since C is upper hemicontinuous and compact valued and A is a Hausdorff space (as it is a metric space), Graph(C) is a closed subset of S × A [24, Theorem 17.10]. As discussed in [38] for all states s ∈ S and actions a ∈ A. To see thatφ is upper semicontinuous, consider any real number c ∈ R. If c > −M , then ∞)). Since ϕ is upper semicontinuous, there is a closed set Fc ⊆ S × A satisfying ϕ −1 ([c, ∞)) = Graph(C) ∩ Fc, makingφ −1 ([c, ∞)) a closed subset of S × A. Otherwise, if c ≤ −M , thenφ −1 ([c, ∞)) = S × A, which is a closed set. Consider a monotonically nonincreasing sequence of bounded realvalued continuous functions {hn ∈ C 0 (S × A) : n ∈ N} converging pointwise toφ; such a sequence is guaranteed to exist [1, §2.4]. For each n ∈ N, define the correspondence Cn : S → P(A) as: Cn(s) = {a ∈ A : ρ C (s, a) < 1/n and gϕ(s) < hn(s, a) + 1/n}, for all states s ∈ S. Now, for any state s ∈ S, recall that the ssection (ρ C )s is continuous. For any n ∈ N, since hn is continuous, the corresponding s-section (hn)s is continuous. Thus, Cn(s) is an open subset of A, and can be represented as: Since the sequence of functions {hn : n ∈ N} is monotonically nonincreasing, these correspondences satisfy C n+1 (s) ⊆ Cn(s) for all states s ∈ S and n ∈ N. For any state s ∈ S and any action a ∈ Cϕ(s), we have ρ C (s, a) = 0 < 1/n and gϕ(s) = ϕ(s, a) ≤ hn(s, a) < hn(s, a) + 1/n for all n ∈ N. This implies Cϕ(s) ⊆ ∞ n=1 Cn(s) for all states s ∈ S. Next, note that gϕ is a measurable function as it is bounded and upper semicontinuous. For any action a ∈ A, recall that the asection (ρ C ) a is a measurable function. Additionally, for any n ∈ N, since hn is continuous, the corresponding a-section (hn) a is a measurable function (as it is continuous). This means the following set is measurable: Lastly, for any state s ∈ S, consider a sequence of actions {an ∈ A : n ∈ N} satisfying an ∈ Cn(s) for all n ∈ N. For each n ∈ N, since ρ C (s, an) < 1/n, there is an action a n ∈ C(s) satisfying ρ A (a n , an) < 2/n. Since C(s) is compact, the sequence of actions {a n ∈ C(s) : n ∈ N} has a limit point a * ∈ C(s). That is, there is a monotonically increasing sequence {n k ∈ N : k ∈ N} such that the subsequence {a n k ∈ C(s) : k ∈ N} converges to a * . Since ρ A (a n k , an k ) < 2/n k ≤ 2/k for all k ∈ N, the subsequence {an k ∈ A : k ∈ N} also converges to a * . For any k, l ∈ N with k > l, we have: hn l (s, an k ) ≥ hn k (s, an k ) > gϕ(s) − 1/n k ≥ gϕ(s) − 1/k, (58) and since hn l is continuous, this means lim k→∞ hn l (s, an k ) = hn l (s, a * ) ≥ gϕ(s). Since the sequence of functions {hn : n ∈ N} converges pointwise toφ from above, we have lim l→∞ hn l (s, a * ) = ϕ(s, a * ) = ϕ(s, a * ) ≥ gϕ(s), implying a * ∈ Cϕ(s). That is, the sequence of actions {an : n ∈ N} has a limit point in Cϕ(s). By the measurability criterion in [1, §2.6], the correspondence Cϕ satisfies the assumptions of [1, Theorem 2.6.B]; therefore, there exists a measurable selection πϕ : S → A of Cϕ, implying πϕ ∈ Π and ϕ(s, πϕ(s)) = gϕ(s) for all states s ∈ S.
Equipped with Lemma 3, we only need to show that the suprema in (48) are equal. Fix a bounded measurable function g : S → R and a state s ∈ S. For any policy π ∈ Π, we have: [Tπ(g)](s) = r(s, π(s)) + γg(f (s, π(s))) ≤ sup a∈C(s) (r(s, a) + γg(f (s, a))). (59) Conversely, for any action a ∈ C(s), we have: r(s, a) + γg(f (s, a)) = rπ s,a (s) + γg(fπ s,a (s)) = [Tπ s,a (g)](s) ≤ sup π∈Π [Tπ(g)](s), where πs,a ∈ Π satisfies πs,a(s) = a. Considering all policies in (59) and all actions in (60) establishes the equality of the suprema. Since r and g are bounded and upper semicontinuous and f is continuous, the function r+γg •f is bounded and upper semicontinuous, so applying Lemma 3 to this function allows us to conclude that T (g) ∈ C u b (S). To show that the MPOP is well-posed, we show that the optimal value function V * is the fixed point of T ; that is T (V * ) = V * . However, though T is a γ-contraction [3, §1.4.1], we cannot immediately apply the contraction mapping principle [24,Theorem 3.48] to show the existence of a unique fixed point of T since C u b (S) is not a closed subset of L 0 b (S). Instead, we consider the sequence of bounded upper semicontinuous function {Vn ∈ C u b (S) : n ∈ N} generated by value iteration when the initial guess is chosen as the constant function V 0 = M/(1 − γ), where M ∈ R + denotes the bound on r. To verify that the functions in this sequence are bounded and upper semicontinuous, note that the initial guess is bounded and upper semicontinuous, and by induction, the value iteration update can be written as: for all n ∈ Z + . The sequence of functions is also monotonically nonincreasing. To see this, note that: for all states s ∈ S. Since T preserves ordering [3], if for some n ∈ Z + we have V n+1 (s) ≤ Vn(s) for all states s ∈ S, then [T (V n+1 )](s) ≤ [T (Vn)](s) for all states s ∈ S. Equivalently, V n+2 (s) ≤ V n+1 (s) for all states s ∈ S. By induction, it follows that the sequence of functions is monotonically nonincreasing. Finally, the sequence of functions is bounded by M/(1 − γ). By definition, V 0 satisfies this bound, and since the sequence of functions is nonincreasing, it is upper bounded by M/(1 − γ). To show the lower bound of −M/(1 − γ), note that: for all states s ∈ S. Similarly, if for some n ∈ Z + we have Vn(s) ≥ −M/(1 − γ) for all states s ∈ S, then: for all states s ∈ S. Therefore, the sequence of functions is bounded below by induction. The pointwise limit of a monotonically nonincreasing sequence of bounded upper semicontinuous functions is upper semicontinuous [1, §2.4], and since the sequence is bounded, so is the limit. Denote the limit by g * ∈ C u b (S). To show that g * is a fixed point of T , note that: