{'fraction_non_alphanumeric': 0.020454545454545454, 'fraction_numerical': 0.0011363636363636363, 'mean_word_length': 5.114583333333333, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Leveraging Anytime Classifiers to solve POMDPs

In the spirit of RL-Classifier reductions [1], we present a method to reduce a specific class of POMDPs to a time-sensitive family of classifiers. POMDPs are in general intractable to solve exactly, so this reduction allows us to solve the full POMDP by using algorithms like Q-learning, which are known to have polynomial convergence time. A great advantage of this approach is that we can exploit classifiers to discretize and approximate the otherwise continuous belief state. The family of classifiers we introduce are so-called anytime classifiers, which guarantee increasing accuracy (decreasing error rate) when given more training data. In real-time reinforcement learning, speed is important, so it is necessary to determine the optimal time to stop training the anytime classifier. This is called the optimal stopping time. We present an algorithm to learn the optimal stopping time for a given anytime classifier and show its connections to neural mechanisms of sensory decision-making. The type of POMDP which can be reduced to an anytime classifier has states which correspond directly to actions augmented by two more actions: continuing and stopping training of the state classifier. The parallels to optimal stopping problems are clear, and we use an extension of Tsitsiklis & Van Roy’s work [2] to derive a learning algorithm for stopping time similar to Q learning. Because all classifiers which are based on calculating the posterior probability using Bayesian models are anytime classifiers (which we show using the Laplace Method), solving this class of POMDP can also be viewed as Bayesian Inference with a stopping rule, with the posterior distribution being parametrized by stopping time.