# How to swap variables in a conditional normal distribution?

I assume that I have two normal distributed variables where one depends on the other:

$$P(A) \sim N(0,\sigma_a)$$

$$P(B|A) \sim N(q\cdot A, \sigma_b)$$

How can I get the reverse $$P(A|B)$$ assuming that B now follows a normal $$P(B)\sim N(\mu_b, \sigma_b)$$ with given mean?

EDIT: The above is something like a structural equation and my model assumes $$\overline b=q \cdot \overline a$$. Now I imagine that I got samples from this process and I select a subsample such that $$B$$ is normal distributed with a given mean. I suppose in that case $$A$$ will have a different mean (and std.dev.?), and I'm interested how to properly show what it is.

It is not appropriate to assume the marginal distribution of $$B$$, since its form follows logically from your specified distributions for $$A \sim \text{N}$$ and $$B|A \sim \text{N}$$. Nevertheless, the form you have assumed is indeed the form that is implied by these distributions. You can find the marginal distribution of interest by writing out the joint density and completing the square to frame it with a single quadratic in the argument $$b$$. This will give you the parameter values of interest.

• I added some clarification to explain what I imagined the setup to be. – Gerenuk Dec 4 '18 at 13:21
• Now you are adding new notation that is left undefined, and making further assumptions that are inconsistent with the previous model framework. That does not help. – Ben Dec 4 '18 at 21:30

I will use variances instead of standard deviations in notation for random variables.

For any two random variables $$A$$ and $$B$$ with respective means $$\mu_A, \mu_B$$ and respective variances $$\sigma_A^2, \sigma_B^2$$, what is the "best" estimate of $$B$$ in terms of $$A$$? By "best" we mean that if $$g(A)$$ denotes the estimate of $$B$$ in terms of $$A$$, then we seek the function $$g(\cdot)$$ such that $$E[(B-g(A))^2]$$, the mean-square error of the estimate, is as small as possible. It is well-known that the minimum mean-square error (MMSE) estimator of $$B$$ in terms of $$A$$ is the conditional mean $$E[B\mid A]$$ of $$B$$ given $$A$$. This conditional mean is a random variable and it is a function of $$A$$, not of $$B$$ as it seems, that is, $$E[B\mid A]$$ is the $$g(A)$$ that we seek. It is also known that $$E[g(A)] = E\big[E[B\mid A]\big]$$, the mean of this particular function of $$A$$, happens, by a miracle of modern mathematics, to equal $$E[B] = \mu_B$$, the mean of $$B$$.

A more constrained version of the above function optimization problem asks, "What is the linear minimum mean-square error (LMMSE) estimator of $$B$$ in terms of $$A$$" where we seek estimators that are constrained to be of the form $$\alpha A + \beta$$ and we try to find $$\alpha$$ and $$\beta$$ such that $$E[(B-(\alpha A + \beta))^2]$$ is as small as possible. Here the well-known answer is that $$\alpha = \rho\frac{\sigma_B}{\sigma_a}, \quad \beta = \mu_B - \rho\frac{\sigma_B}{\sigma_a}\mu_A = \mu_B - \alpha\mu_A \tag{1}$$t where $$\rho$$ is the correlation coefficient, and it is also known that this function $$\alpha A + \beta$$ has mean and variance given by $$E[\alpha A + \beta] = \alpha \mu_A + \beta = \mu_B, \quad \operatorname{var}(\alpha A + \beta) = \sigma_B^2(1-\rho)^2.\tag{2}$$

As pointed out by @Ben, your assumptions that $$A$$ is a normal random variable $$N(0, \sigma_A^2)$$ and that the conditional distribution of $$B$$ given that $$A$$ has value $$a$$ is $$N(qa,\sigma_b^2)$$ (where $$q$$ and $$\sigma_b$$ are known constants (not dependent on $$a$$ in any way)) is equivalent to the assumption that $$A$$ and $$B$$ are jointly normal random variables. If you don't like this bald assertion, notice that \begin{align} f_A(a) &\propto \exp\left(-\frac{a^2}{2\sigma_A^2}\right)\\ f_{B\mid A}(b\mid a) &\propto \exp\left(-\frac{(b-qa)^2}{2\sigma_b^2}\right) \end{align} and so the joint density $$f_{A,B}(a,b) = f_{B\mid A}(b\mid a)\cdot f_A(a) \propto \exp(Q(a,b))$$ where $$Q(a,b)$$ is a quadratic function of $$a$$ and $$b$$, that is, after completing the square etc, it will be found that the joint density is a jointly normal (a.k.a. bivariate normal) density.

Now, in the special case when $$A$$ and $$B$$ are jointly normal random variables (that is, they have a bivariate normal density), the MMSE estimator $$E[B\mid A]$$ is a linear function of $$A$$, and so the MMSE estimator must coincide with the LMMSE estimator, no? For jointly normal random variables, the conditional distribution of $$B$$ given $$A = a$$ is a normal distribution with mean and variance given by $$E[B \mid A = a] = \rho\frac{\sigma_B}{\sigma_A}a + \mu_B - \rho\frac{\sigma_B}{\sigma_a}\mu_A, \quad \operatorname{var}(B \mid A = a) = \sigma_B^2(1-\rho^2) \tag{3}$$ where we are told that $$\mu_A = 0$$ in this instance, and that $$E[B \mid A = a] = qa, \quad \operatorname{var}(B \mid A = a) = \sigma_b^2\tag{4}$$ (Note the subscript is a lower-case $$b$$). Comparing $$(3)$$ and $$(4)$$ we see that it must be that $$\mu_B = 0$$, and that the known $$q$$ equals $$\rho\frac{\sigma_B}{\sigma_A}$$ (remember that we are given the value of $$\sigma_A$$) while the known $$\sigma_b^2$$ must equal $$\sigma_B^2(1-\rho^2)$$. So we get that $$\rho\sigma_B = q\sigma_A, \sigma_b^2 = \sigma_B^2-\rho^2\sigma_B^2 = \sigma_B^2 - (q\sigma_A)^2\\ \implies \sigma_B^2 = \sigma_b^2 + (q\sigma_A)^2, \quad \rho = \frac{q\sigma_A}{\sqrt{\sigma_b^2 + (q\sigma_A)^2}} \tag{5}$$

So, $$A$$ and $$B$$ are zero-mean jointly normal random variables with variances $$\sigma_A^2$$ and $$\sigma_B^2 = \sigma_b^2 + (q\sigma_A)^2$$ and correlation coefficient $$\rho = \frac{q\sigma_A}{\sqrt{\sigma_b^2 + (q\sigma_A)^2}}$$. It follows that the conditional distribution of $$A$$ given that $$B = b$$ is a normal distribution with mean and variance $$E[A \mid B = b] = \rho\frac{\sigma_A}{\sigma_B}b, \quad \operatorname{var}(A \mid B = b) = \sigma_A^2(1-\rho^2)$$
The text passage primarily discusses the problem of finding the conditional distribution \( P(A|B) \) given the distributions \( P(A) \) and \( P(B|A) \). It also includes a detailed explanation of the relationship between the variables and the properties of jointly normal random variables.
single_question_with_answer

1. the passage contains a mathematical question about how to find the reverse conditional distribution \( p(a|b) \) given the distributions \( p(a) \) and \( p(b|a) \).
2. the passage provides a detailed answer, including the derivation of the mean and variance of \( p(a|b) \) using properties of jointly normal random variables.
Given a text passage, determine:
    1. If it contains a mathematical question (including word problems or calculations)
    2. If an answer to that mathematical question is provided in the text

First, describe the content structure you found in 1-2 sentences.

Then provide your analysis in the following format:
TYPE: <one of the following values>
- NO_QUESTION
- SINGLE_QUESTION
- SINGLE_QUESTION_WITH_ANSWER
- MULTIPLE_QUESTIONS
- MULTIPLE_QUESTIONS_WITH_ANSWERS
system
# How to swap variables in a conditional normal distribution?

I assume that I have two normal distributed variables where one depends on the other:

$$P(A) \sim N(0,\sigma_a)$$

$$P(B|A) \sim N(q\cdot A, \sigma_b)$$

How can I get the reverse $$P(A|B)$$ assuming that B now follows a normal $$P(B)\sim N(\mu_b, \sigma_b)$$ with given mean?

EDIT: The above is something like a structural equation and my model assumes $$\overline b=q \cdot \overline a$$. Now I imagine that I got samples from this process and I select a subsample such that $$B$$ is normal distributed with a given mean. I suppose in that case $$A$$ will have a different mean (and std.dev.?), and I'm interested how to properly show what it is.

It is not appropriate to assume the marginal distribution of $$B$$, since its form follows logically from your specified distributions for $$A \sim \text{N}$$ and $$B|A \sim \text{N}$$. Nevertheless, the form you have assumed is indeed the form that is implied by these distributions. You can find the marginal distribution of interest by writing out the joint density and completing the square to frame it with a single quadratic in the argument $$b$$. This will give you the parameter values of interest.

• I added some clarification to explain what I imagined the setup to be. – Gerenuk Dec 4 '18 at 13:21
• Now you are adding new notation that is left undefined, and making further assumptions that are inconsistent with the previous model framework. That does not help. – Ben Dec 4 '18 at 21:30

I will use variances instead of standard deviations in notation for random variables.

For any two random variables $$A$$ and $$B$$ with respective means $$\mu_A, \mu_B$$ and respective variances $$\sigma_A^2, \sigma_B^2$$, what is the "best" estimate of $$B$$ in terms of $$A$$? By "best" we mean that if $$g(A)$$ denotes the estimate of $$B$$ in terms of $$A$$, then we seek the function $$g(\cdot)$$ such that $$E[(B-g(A))^2]$$, the mean-square error of the estimate, is as small as possible. It is well-known that the minimum mean-square error (MMSE) estimator of $$B$$ in terms of $$A$$ is the conditional mean $$E[B\mid A]$$ of $$B$$ given $$A$$. This conditional mean is a random variable and it is a function of $$A$$, not of $$B$$ as it seems, that is, $$E[B\mid A]$$ is the $$g(A)$$ that we seek. It is also known that $$E[g(A)] = E\big[E[B\mid A]\big]$$, the mean of this particular function of $$A$$, happens, by a miracle of modern mathematics, to equal $$E[B] = \mu_B$$, the mean of $$B$$.

A more constrained version of the above function optimization problem asks, "What is the linear minimum mean-square error (LMMSE) estimator of $$B$$ in terms of $$A$$" where we seek estimators that are constrained to be of the form $$\alpha A + \beta$$ and we try to find $$\alpha$$ and $$\beta$$ such that $$E[(B-(\alpha A + \beta))^2]$$ is as small as possible. Here the well-known answer is that $$\alpha = \rho\frac{\sigma_B}{\sigma_a}, \quad \beta = \mu_B - \rho\frac{\sigma_B}{\sigma_a}\mu_A = \mu_B - \alpha\mu_A \tag{1}$$t where $$\rho$$ is the correlation coefficient, and it is also known that this function $$\alpha A + \beta$$ has mean and variance given by $$E[\alpha A + \beta] = \alpha \mu_A + \beta = \mu_B, \quad \operatorname{var}(\alpha A + \beta) = \sigma_B^2(1-\rho)^2.\tag{2}$$

As pointed out by @Ben, your assumptions that $$A$$ is a normal random variable $$N(0, \sigma_A^2)$$ and that the conditional distribution of $$B$$ given that $$A$$ has value $$a$$ is $$N(qa,\sigma_b^2)$$ (where $$q$$ and $$\sigma_b$$ are known constants (not dependent on $$a$$ in any way)) is equivalent to the assumption that $$A$$ and $$B$$ are jointly normal random variables. If you don't like this bald assertion, notice that \begin{align} f_A(a) &\propto \exp\left(-\frac{a^2}{2\sigma_A^2}\right)\\ f_{B\mid A}(b\mid a) &\propto \exp\left(-\frac{(b-qa)^2}{2\sigma_b^2}\right) \end{align} and so the joint density $$f_{A,B}(a,b) = f_{B\mid A}(b\mid a)\cdot f_A(a) \propto \exp(Q(a,b))$$ where $$Q(a,b)$$ is a quadratic function of $$a$$ and $$b$$, that is, after completing the square etc, it will be found that the joint density is a jointly normal (a.k.a. bivariate normal) density.

Now, in the special case when $$A$$ and $$B$$ are jointly normal random variables (that is, they have a bivariate normal density), the MMSE estimator $$E[B\mid A]$$ is a linear function of $$A$$, and so the MMSE estimator must coincide with the LMMSE estimator, no? For jointly normal random variables, the conditional distribution of $$B$$ given $$A = a$$ is a normal distribution with mean and variance given by $$E[B \mid A = a] = \rho\frac{\sigma_B}{\sigma_A}a + \mu_B - \rho\frac{\sigma_B}{\sigma_a}\mu_A, \quad \operatorname{var}(B \mid A = a) = \sigma_B^2(1-\rho^2) \tag{3}$$ where we are told that $$\mu_A = 0$$ in this instance, and that $$E[B \mid A = a] = qa, \quad \operatorname{var}(B \mid A = a) = \sigma_b^2\tag{4}$$ (Note the subscript is a lower-case $$b$$). Comparing $$(3)$$ and $$(4)$$ we see that it must be that $$\mu_B = 0$$, and that the known $$q$$ equals $$\rho\frac{\sigma_B}{\sigma_A}$$ (remember that we are given the value of $$\sigma_A$$) while the known $$\sigma_b^2$$ must equal $$\sigma_B^2(1-\rho^2)$$. So we get that $$\rho\sigma_B = q\sigma_A, \sigma_b^2 = \sigma_B^2-\rho^2\sigma_B^2 = \sigma_B^2 - (q\sigma_A)^2\\ \implies \sigma_B^2 = \sigma_b^2 + (q\sigma_A)^2, \quad \rho = \frac{q\sigma_A}{\sqrt{\sigma_b^2 + (q\sigma_A)^2}} \tag{5}$$

So, $$A$$ and $$B$$ are zero-mean jointly normal random variables with variances $$\sigma_A^2$$ and $$\sigma_B^2 = \sigma_b^2 + (q\sigma_A)^2$$ and correlation coefficient $$\rho = \frac{q\sigma_A}{\sqrt{\sigma_b^2 + (q\sigma_A)^2}}$$. It follows that the conditional distribution of $$A$$ given that $$B = b$$ is a normal distribution with mean and variance $$E[A \mid B = b] = \rho\frac{\sigma_A}{\sigma_B}b, \quad \operatorname{var}(A \mid B = b) = \sigma_A^2(1-\rho^2)$$
user
The text passage primarily discusses the problem of finding the conditional distribution \( P(A|B) \) given the distributions \( P(A) \) and \( P(B|A) \). It also includes a detailed explanation of the relationship between the variables and the properties of jointly normal random variables.

TYPE: SINGLE_QUESTION_WITH_ANSWER

1. The passage contains a mathematical question about how to find the reverse conditional distribution \( P(A|B) \) given the distributions \( P(A) \) and \( P(B|A) \).
2. The passage provides a detailed answer, including the derivation of the mean and variance of \( P(A|B) \) using properties of jointly normal random variables.