{'fraction_non_alphanumeric': 0.047416612164813604, 'fraction_numerical': 0.005886200130804447, 'mean_word_length': 4.592321755027422, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A NEW MESSAGE LENGTH APPROXIMATION FOR PARAMETER ESTIMATION AND MODEL SELECTION

This paper examines Bayesian two-part coding schemes as tools for parameter estimation and model selection. The Wallace–Freeman message length approximation to strict minimum message length can be used to obtain two-part message lengths. However, this approximation relies on some strong assumptions regarding the likelihood function and prior distribution which do not hold for a large range of models. We present a new two-part message length formula that is more widely applicable than the popular Wallace–Freeman message length approximation, while remaining significantly easier to compute than the exact strict minimum message length procedure. 1. MML TWO-PART CODES Consider the problem of choosing a plausible explanation for some observed data y = (y1, . . . , yn) ∈ Y ⊆ R. The possible explanations are the distributions, or (fully specified) models, contained in a countable set of parametric model structures γ ∈ Γ. Let pγ(y|θ) denote the model1, in model structure γ, indexed by θ ∈ Θγ ⊆ R. The minimum encoding approach [1, 2] to inference suggests that the model that most compresses the data is the most plausible explanation. One way to compress the data is by two-part coding, in which the model and the data are compressed together as a two-part message. This idea is central to the minimum message length principle (MML). The MML principle is explicitly Bayesian in nature, so we further assume that a suitable prior distribution, πγ(θ), θ ∈ Θγ exists for all γ ∈ Γ. The first part of the message, or assertion, states which model, pγ(·|θ), from the structure γ, is to be used to compress the data. The second part, or detail, states the data y using the nominated model from γ. Let the length of these two terms be denoted by I(θ; γ) and I(y|θ; γ), respectively. Further, let I(γ) denote the length of a preamble code stating which structure from Γ is being used. Estimation of both a model structure, as well as the model parameters, may be simultaneously performed by solving { γ̂, θ̂ } = arg min γ∈Γ,θ∈Θγ {I(γ) + I(θ; γ) + I(y|θ; γ)} . 1We acknowledge that this use of the term “model” differs from much of the traditional statistical literature. This is done to keep the terminology in this paper consistent with the MML literature. In the strict MML (SMML) procedure, the assertion and detail codes are constructed so that for a given structure γ, the expected joint codelength is minimised, the expectation being taken with respect to the marginal distribution of the data. The optimisation problem implicit in this minimisation is in general NP-hard [3], and thus the procedure is impractical for all but the simplest of problems. 1.1. The Wallace–Freeman Codelength Under suitable regularity conditions, Wallace and Freeman proposed an approximate codelength formula which we shall refer to as MML87 [4]. For a structure γ with k free parameters, the MML87 assertion and detail lengths for a model θ ∈ Θγ are I87(θ; γ) = − log ( πγ(θ) |Jγ(θ)| 1 2 )