TITLE: Closed form for KL divergence equation
QUESTION [2 upvotes]: For a fixed $p\in(0,1)$ and $y>0$ is it possible to find a closed form for $x\in(0,1)$ such that
$$x\log\left(\frac{x}p{}\right) + (1-x)\log\left(\frac{1-x}{1-p}\right) = y ?$$
I've tried a bunch of calculations but didn't come up with anything, so any hint or help will be very useful!

REPLY [5 votes]: I am skeptical about a possible closed form but, beside numerical methods, I think that decent approximations could be obtained.
Conside that you look for the zeros of function
$$f(x)=x\log\left(\frac{x}p{}\right) + (1-x)\log\left(\frac{1-x}{1-p}\right) - y $$ for which
$$f'(x)=\log \left(\frac{x}{p}\right)-\log \left(\frac{1-x}{1-p}\right)\qquad \text{and} \qquad f''(x)=\frac{1}{x}+\frac{1}{1-x} \quad >0$$
The first derivative cancels at
$$x_*=p \implies f(x_*)=-y$$ For a first approximation, expand $f(x)$ as a Taylor series around $x_*$; this will give
$$f(x)=-y+\frac{(x-p)^2}{2 (1-p) p}+O\left((x-p)^3\right)$$ giving
$$x_0=p\pm\sqrt{2 p(1-p)  y }$$ We also have that
$$f(0)=-y-\log(1-p)\qquad \text{and} \qquad f(1)=-y-\log(p)$$ so, depending on the respective values of $p$ and $y$, we can have $0$, $1$ or $2$ solutions.
Suppose that $f(0) < 0$ leading to a single root. We have that
$$y=\sum_{n=2}^\infty \frac{\left((-1)^n (1-p) p\right)^{-n} \left((-1)^n (1-p) p^n+p (1-p)^n\right) } {n(n-1) } (x-p)^n$$ Truncating to some order and using series reversion
$$x \sim p+t +\frac{1-2 p }{6 [p(1-p)]}t^2-\frac{2p(1-p)+1 }{72 [p(1-p)]^2}t^3+ O(t^4)$$ where $t=\sqrt{2 p(1-p)  y }$.
Trying for $p=\frac 13$ and $y=\frac 12$, this would first give $x_0=\frac{1+\sqrt{2}}{3}=0.804738$, the truncated series $x=\frac{168+131 \sqrt{2}}{432}=0.817736$ while the solution (given using Newton method) is $x=0.818888$.
All of this could be signifiantly improved at the price of a few more terms in the series expansion.
{"set_name": "stack_exchange", "score": 2, "question_id": 4380806}