{'fraction_non_alphanumeric': 0.017204301075268817, 'fraction_numerical': 0.0, 'mean_word_length': 5.1655629139072845, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Single sample path based optimization of Markov systems: examples and algorithms

Motivated by the needs of online optimization of real world engineering systems, we study the single sample path based algorithms for Markov decision problems (MDPs). We give a simple example to explain the advantages of the sample path based approach over the traditional computation based approach: matrix inversion is not required; some transition probabilities do not have to be known; it may save storage space; and it gives the flexibility of iterating the actions for a subset of the state space in each iteration. The effect of the estimation errors and the convergence property of the sample path based approach are studied. Finally, we propose a "fast" algorithm which updates the policy whenever the system reaches a particular set of states; the algorithm converges to the true optimal policy with probability one under some conditions.