I'm not sure whether this counts as a question. However, I'd be happy to receive feedback for the validity of my reasoning.

Recently, I read a bit about Jeffreys' prior and the "problem" with using uniform distributions as non-informative priors.

The way I understand it now is as follows. We seek a distribution of an unknown parameter that can be interpreted as if we "knew nothing" about it (except possibly that is resides in e.g. $[0, 1]$, from now on we shall assume this).

Intuitively, if we "know nothing" about the parameter, it is natural to assume that we also "know nothing" about any monotone transformation of it.

A priori (no pun intended) it is also natural to model ignorance about a parameter by assuming that it follows a uniform distribution.

As it happens, however, these two intuitively reasonable claims cannot, in general, be satisfied simultaneously.

As an example, assume we model the parameter $\theta$, say, of which we "know nothing" as having a uniform distribution on $[0, 1]$.

Now consider the monotonically transformed variable $\eta(\theta) = \frac{e^{10 \cdot (\theta - 1)} - e^{-10}}{1 - e^{-10}}$.

Plotting this function, it is clear that it will have a high density near zero and rather low elsewhere, i.e. the distribution is far from uniform.

But I think this example shows why the intuitively reasonable claim that the concept of "knowing nothing" can be captured by a uniform prior breaks down; using a uniform distribution, we do "know" some stuff, e.g. that the probability of the parameter being larger than 0.99 is low, just one percent.

So thinking about it, it is no big surprise that this is reflected in the derived density of our transformed parameter $\eta$; mapping $[0, 1]$ to itself monotonically such that a set that constitutes a large amount of the total probability is mapped to a set covering only a small portion of $[0, 1]$, of course the implied distribution of $\eta$ cannot be uniform!

Our "knowledge" of $\theta$ implies the "knowledge" that the value of $\eta$ is likely to be low.

This is naturally good to be aware of, but I suppose that whether it is a problem or not is subjective.

  • $\begingroup$ Interesting, but I'm not sure how this relates to Jeffreys' prior. There the idea is that we can only capture "knowing nothing" about a parameter in the form of a prior distribution if we do know something about the role that parameter plays, i.e. how it enters the likelihood. $\endgroup$
    – A. Donda
    Jul 20 '15 at 3:27
  • $\begingroup$ True, it doesn't have anything to do with Jeffrey's prior as such - I just came to think about this when reading a motivation of Jeffrey's prior as a sort of remedy of the above issue with uniforms as non-informative priors. $\endgroup$
    – Christian
    Jul 20 '15 at 7:59

This is a very interesting idea, although I suspect one could take issue with the statement "we know nothing about any monotone transformation of $\theta$." We actually do, we know the functional relationship between that parameter and another which is uniformly distributed over $[0, 1]$. I think what your example shows is that the prior on $\theta$ is a form of knowledge, so our model really is not consistent with the idea of "knowing nothing." If we're being entirely honest, and want to claim absolute ignorance about the underlying model, then it seems we can't even postulate non-informative priors.


Your Answer

