{'fraction_non_alphanumeric': 0.020047169811320754, 'fraction_numerical': 0.0, 'mean_word_length': 5.0212765957446805, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Approximating a non-homogeneous HMM with Dynamic Spatial Dirichlet Process

In this work we present a model that uses a Dirichlet process (DP) with a dynamic spatial constraints to approximate a non-homogeneous hidden Markov model (NHMM). The coefficient of the spatial constraint, which is locally dependent on each site, modulates the time-variant transition probability matrix. In our model, we use the DP in combination with variational Bayesian inference to estimate the local coefficients and the time-dependent structure of the hidden states. In addition, the formulation of the NHMM within the DP framework does not require the specification of the number of states. Our results demonstrate that the proposed model can uncover the hidden states when the observed data is generated by a NHMM model and the number of hidden states is unknown.