{'fraction_non_alphanumeric': 0.030620673939670152, 'fraction_numerical': 0.01492579827384852, 'mean_word_length': 4.597576530612245, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Binary Independent Component Analysis via Non-stationarity

We consider independent component analysis of binary data. While fundamental in practice, this case has been much less developed than ICA for continuous data. We start by assuming a linear mixing model in a continuous-valued latent space, followed by a binary observation model. Importantly, we assume that the sources are non-stationary; this is necessary since any non-Gaussianity would essentially be destroyed by the binarization. Interestingly, the model allows for closed-form likelihood by employing the cumulative distribution function of the multivariate Gaussian distribution. In stark contrast to the continuous-valued case, we prove non-identifiability of the model with few observed variables; our empirical results imply identifiability when the number of observed variables is higher. We present a practical method for binary ICA that uses only pairwise marginals, which are faster to compute than the full multivariate likelihood.


INTRODUCTION
Despite significant progress in both linear and nonlinear ICA in recent years (Hyvärinen and Morioka, 2016;Hyvärinen et al., 2019;Khemakhem et al., 2020), ICA for binary data remains a challenging and important problem. Binary data is abundant in various fields, such as bioinformatics, health informatics, social sciences, natural language, and electrical engineering. An ICA model for binary data may also open new opportunities in solving problems closely related to ICA, such as causal discovery (Shimizu et al., 2006).
Methods for binary ICA have been proposed based on either binary or continuous-valued independent components. In the case of binary components, Himberg and Hyvärinen (2001); Nguyen and Zheng (2011) as-sumed an OR mixture model.Some extensions of Latent Dirichlet can be seen as binary ICA (Podosinnikova et al., 2015;Buntine and Jakulin, 2005). On the other hand, Kabán and Bingham (2006) presented an approach based on a latent linear model and binarized observations, although the components were restricted to the unit interval, which restricts its applicability. Khemakhem et al. (2020) presented a nonlinear ICA model with binarized observations, which is a promising approach for our purposes.
Our goal here is to study the prospects of ICA for binary data by using an intuitively appealing model. Identifiability of such a model is crucial to investigate, and we also want a consistent estimator which is not based on approximations whose validity is not clear. None of the approaches above fulfills all these criteria. 1 We propose a binary ICA model inspired by recent developments in nonlinear ICA. We formulate a latent linear model with a separate binarizing measurement equation. Crucially, we assume the components to be non-stationary, which is a powerful principle and very useful here because any non-Gaussianity may be destroyed by binarization. Thus, we obtain a binary ICA model whose likelihood can actually be described in closed form via the multivariate Gaussian cumulative distribution function. We further propose to combine the likelihood with a moment-matching approach to obtain a fast and accurate estimation algorithm. In fact, due to the model structure, pairwise marginal distributions of non-binarized data can be accurately estimated from binary sample data and the likelihood can be computed directly from them. We investigate the identifiability of the model, and somewhat surprisingly, we show that low-dimensional models are in fact non-identifiable-while higher-dimensional models are (empirically) shown to be identifiable.

A MODEL FOR BINARY ICA
We define here a binary counterpart of the linear ICA model. In particular, we consider here a model with non-stationarity: we assume the data consists of n observed variables, divided into n u segments which express the non-stationarity. Thus, each data point has a segment index u assigned to it. This additionally observed variable u assumes categorical values, which is a special case of the auxiliary variable framework of Khemakhem et al. (2020). Such non-stationarity based on a segment-wise Gaussian model is well-known in linear ICA (Pham and Cardoso, 2001;Miettinen et al., 2017). It is not only natural in the case of non-stationary time series, but also when there is any other external discrete variable, such as the experimental condition or intervention, or even a class label which modulates the distribution of the data.
We assume the data is generated from n z latent variables (independent components, or sources), collected into a latent random vector z, which are generated independently of each other from a Gaussian distribution. Crucially, the parameters of the Gaussian distribution change as a function of the segment as where Σ u z is a diagonal matrix of the source variances in segment u.
We define "intermediate" variables y which are a linear mixing of the sources by a mixing matrix A with n rows and n z linearly independent columns y = Az ∼ N (Aµ u z , AΣ u z A ) While some work in ICA considers noisy continuous observations by adding noise to y, we can consider here binarized observations instead. Binarization is done using a linking function σ so that We use a linking function based on the Gaussian cumulative distribution function (CDF): where Φ is the CDF of Gaussian distribution, here with mean 0 and variance 1. We use π 8 as the coefficient to match closely to the often employed sigmoid function σ(y i ) = 1 1+e −y i (Waissi and Rossin, 1996;Li, 2021). This Gaussian CDF-based linking function has certain nice algebraic properties compared to the sigmoid, as we will see in Section 3. Furthermore, the linking function has the following intuitive interpretation. Take y i , add independent noise form N (0, 8 π ), and binarize y i simply by a hard threshold 0 to get x i . This gives the same distribution for x i , since the probabilities match: thus consists of the following parameters: mixing matrix A, the means µ u z and the diagonal (co)variance matrices Σ u z for all segments u, denoted by {µ u z } u and {Σ u z } u .

THE LIKELIHOOD
A surprising observation regarding the the latent variable model defined in Section 2, is that we can calculate the likelihood in closed-form by employing the multivariate Gaussian CDF. For example, the model defines the probability of the data vector of all ones, denoted by 1, as: where the univariate Gaussian CDFs are written as a multivariate Gaussian CDF Φ with a identity covariance matrix. The benefit of using a Gaussian CDF based linking function comes into play here, as the intergral is directly a value of a multivariate Gaussian CDF (Waissi and Rossin, 1996;Li, 2021): The above integral actually specifies the probability of first drawing y and then, independently, drawing standard Gaussian variable n ∼ N (0, I) that is elementwise smaller. We therefore have: This motivates us to define q, an important quantity in the following developments as: which is simply a noisy, rescaled and sign-flipped version of the linear mixture y. As noted in the preceding section, using the Gaussian CDF linking function is equivalent to adding Gaussian noise and thresholding, which is exactly what happens here, and very useful since then in each segment, we have a purely Gaussian generative model which is thresholded. In fact, since q is the sum of two independent Gaussian random vectors it also has a Gaussian distribution Figure 1: Binary ICA model for two observed variables and three segments. For each segment there is a bivariate Gaussian distribution on q, the probability of an assignment to the binary observed variables is the probability mass in the corresponding quadrant.
The probability of data vector of ones is then: where Φ is the CDF of the multivariate Gaussian such that all variables integrated from −∞ to 0; it is readily implemented in basic packages (Genz and Bretz, 2009).
Similar derivation gives the probabilities for other assignments to x. These probabilities can be expressed compactly for all value assignments as: in which the multivariate Gaussian probability density function is integrated from the lower bound l(x u ) to the upper bound u(x u ) with: Importantly, this formulation allows for a particularly clear intuitive interpretation of the model. Figure 1 shows this for two observed variables and three segments. For each segment the model defines a bivariate Gaussian distribution for q, depicted by colors and contours on the planes. The probability for an assignment of the observed binary variables in a segment is simply the probability mass in a corresponding quadrant. The multivariate Gaussian distributions for q in each segment are related in the sense that they are formed by the same mixing matrix performing on independent sources particular to the segment.
The log-likelihood of the whole data set can then be calculated as where c(x, u) is the count of the data points x, u and the sum is taken over all possible assigments to x, u.

ON IDENTIFIABILITY
Many ICA models can only be identified up to scaling and permutation indeterminacies of the sources (Hyvärinen et al., 2001;Khemakhem et al., 2020). Straightforwardly we can see that those limitations apply for our model as well. By re-ordering columns of the mixing matrix and the source, the implied distribution is unaffected; similarly, we can counteract the scaling (or sign-flip) of the mixing matrix columns by scaling (or sign-flipping) the sources. However, binarization actually induces additional indeterminacies as we will show next.

The Binarization Indeterminacy
First, we note that the probability in Equation 4, stays exactly the same even if q is multiplied by a diagonal matrix Q u , possibly different for each segment, with positive entries on the diagonal: Note that this is valid even if the elementwise operator is > or a mixture of > and <. 2 That is, we lose the scale information on q in the binarization. Two binary ICA models there are positive diagonal matrices {Q u } u such that for each segment u means and covariances of q satisfy: which can be written using the model parameters (Equations 2 and 3) as: That is because the mass of both distributions in each of the 4 quadrants is identical. Figure 2 shows an example of this equivalence relation for one segment and two observed variables. The two Gaussian distributions represented by the blue and red contours imply the exact same joint distribution for binary observed variables. The amount of mass in each of the 4 quadrants is exactly the same.

Row Order Indeterminacy for n = 2
One of consequences of the binarization indeterminacy is the following non-identifiability result, proven in Appendix A.
Theorem 1. If the row order of the mixing matrix A of a two-dimensional binary ICA model is reversed, then the source means µ u z and variances Σ u z can be adjusted such that the implied (observed) binary distributions remains identical.
Although the result may generalize to certain sparse higher dimensional models, fortunately, it does not seem to jeopardize the estimation of higher dimensional models in general.
The result does have serious consequences for causal discovery (Shimizu et al., 2006). Consider two structural equation models, implying opposite causal directions: where z has a Gaussian distribution in each segment with diagonal covariance matrix Σ u z . The models imply respectively the mixing models: If we observed binarized y, i.e. x, we can at most identify the mixing matrix up to row order, column order and column scale. By switching the column order and then the row order of mixing matrix on the left, we get the mixing matrix on the right. Thus, unlike in the continuous case, we cannot detect the causal direction between two variables without further assumptions.

Identifiability of q-Correlations
Note that the indistinguishable models in Equation 8 have equal correlation matrices (i.e. matrices of Pearson correlation coefficients) for the random variables q. The next theorem and corollary show that the correlations between elements of q are indeed theoretically identifiable from the distributions of the binary observed variables. Intuitively, the higher the correlation the more likely will the pair of binary observed variables receive equal assignments. The fairly technical proof is given in Appendix B.
Theorem 2. Two binary ICA models imply different binary distributions (in a given segment) if the correlation matrices for q are not equal.
This result is crucial for the development of our novel estimation method (Section 5.2), via the corollary: Corollary 1. The correlation matrix of q in a given segment is identifiable from binary observations.
On the other hand, the following theorem recaps the well-known result (Hyvärinen et al., 2001;Pham and Cardoso, 2001) that the means do not help in estimating the mixing matrix: Theorem 3. If two models M and M with n = n z imply the same correlation matrices for q (in a given segment) then the means µ u z can be adjusted such that the implied binary distributions are identical.

A Heuristic Variable-Counting Approach
-10 n= 3 -9 -9 -9 -9 -9 n= 4 -12 -10 -8  We conclude our identifiability analysis by a wellknown heuristic approach to identifiability used in factor analysis. It is based on counting the number of statistics we can calculate (or equations that we can formulate), and the number of unknowns (model parameters) we need to solve. If the number of statistics is at least as large as the number of parameters, there is hope that the model is identifiable. The calculations in Table 1 are based on Equations 9 and 10 when the number of sources equals the number of observations (n = n z ); recall that the number of segments is denoted by n u . The equations/statistics correspond to n u (n 2 − n)/2 correlations, n u · n variances and n u · n means. Unknowns include n · n mixing matrix coefficients, n u · n segment-wise source variances, n u · n segment-wise source means, as well as n u · n segmentwise scaling terms (positive diagonal elements of Q u ). 3 These calculations suggest how identifiability depends on the numbers of the segments and the observed variables. We investigate the validity of these predictions experimentally below. Interestingly, the calculations suggest that the bivariate case is never identifiable.

METHODS FOR BINARY ICA
Next, we present three methods for estimating the binary ICA model, building on the theory in Sections 3 and 4. The BLICA method of Section 5.2 is the main novel contribution of the paper.

Maximum Likelihood Estimation
We have already derived the likelihood of our model. A straightforward approach is then to optimize this using e.g. L-BFGS. The gradient involves the calculation of moments for the truncated multivariate Gaussian distribution. These can be obtained from R-package tmvtnorm (Wilhelm and G, 2015). Unfortunately, the computation of the likelihood and its gradient can only be done for small models in practice, because the evaluation of multivariate Gaussian CDF is time consuming, necessitating sampling-based approximations.
Our experiments refer to this as full MLE.

The BLICA Method
However, we can circumvent computational burden of the high dimensional Gaussian CDF. Due to the theory in Section 4, the correlations of q convey the essential information between the binary data and the continuous mixing model. Since the marginalization properties of our model are inherited from the multivariate Gaussian, such correlations can be estimated Estimate the correlation between q i and q j by maximizing the marginal pairwise likelihood of x i and x j (in segment u).

5:
Form and regularize the correlation matrixΣ u q obtained from the pairwise correlations. 6: Optimize scaled Gaussian likelihood with L-BFGS over sufficient statisticsΣ u q from all segments u. 7: Return the estimated mixing matrix A and source variances Σ u z for all segments u.
from pairwise marginal distributions; in 2D the Gaussian CDF is still quite quick to compute. Thus, we combine maximum likelihood estimation with what could be called a "moment-matching" approach as follows. We first recover the pairwise correlations of the continuous-valued q from the observed binary data (this is possible by Corollary 1) via MLE in 2D. Then we fit those correlations to the correlations implied by the latent linear mixing model using a more scalable MLE in the continuous-valued latent space. The resulting algorithm is summarized as Algorithm 1.
Correlation estimation. On line 4, we estimate each correlations separately by directly fitting a model for q in Equation 6 in two dimensions. To calculate multivariate Gaussian CDF we use the R package mvtnorm (Genz and Bretz, 2009). We employ the GenzBretz method, which is particularly suitable for the fast evaluation needed here (Genz, 1993). Furthermore, the estimation can be simplified (Lee and Sompolinsky, 1999). Due to Equation 10 the variances Σ q [i, i] can be set to 1. Furthermore, since the marginal of x i is can be computed using the inverse CDF (Genz and Bretz, 2009). The univarite optimization problem in the interval [−1, 1] can be solved using a line search method (Brent, 2013). The scalability of Algorithm 1 depends crucially on this step as n u · (n 2 − n)/2 correlations need to be estimated.
Regularization. When estimating the correlation of q from sample data, it can happen that the correlation matrix C is close to singular or not positive definite.
We use the following regularization on line 5 based a parameter r (Warton, 2008), which marks the approximate condition number targeted. The regularized correlation matrix is then where λ 1 is the largest and λ n the smallest eigenvalue ofΣ u q . This regularization keeps the unit diagonal. Scaled Gaussian likelihood. Finally on line 6, we fit the estimated correlations using the Gaussian likelihood model over the different segments and a stationary mixing matrix as defined by the model (Section 2). But in contrast to the usual case where we have covariance matrices, we need to here account for the "binarization indeterminacy", resulting in additional nuisance scaling parameter, as pointed out above. We use the term scaled Gaussian likelihood to refer to ordinary multivariate Gaussian likelihood where we include additional parameters Q u as the scaling factors. The fitting is thus done by the following scaled Gaussian likelihood based on the sufficient statistics: where recall that Σ u q = Q u (I + AΣ u z A T )Q u by Equation 10 is a function of the mixing matrix A, source variances {Σ u z } u (diagonal, positive elements) and scaling factors {Q u } u (diagonal, positive elements). Note that without the scaling factors {Q u } u , the mixing matrix A could be found via joint diagonalization (Miettinen et al., 2017). Note also that due to Theorem 6 the source means need not be estimated since only correlations can and need to be estimated. Here, instead we perform the fitting by maximizing this likelihood using L-BFGS (Liu and Nocedal, 1989) with respect to the aforementioned parameters.

Binary ICA through Linear iVAE
Khemakhem et al. (2020) presented the identifiable Variational Autoencoder (iVAE), an approach for nonlinear ICA employing variational autoencoders (Kingma and Welling, 2014;Rezende et al., 2014) that assumes access to an additionally observed variable such that the sources are independent given the aux-iliary variable; further, each source follows an exponential family distribution given the auxiliary variable. Here, we apply iVAE to estimate the binary ICA model from Section 2. As proposed by Kingma and Welling (2014) and Khemakhem et al. (2020), we use the factorized Bernoulli observational model and apply a sigmoid function element-wise to the output of the decoder to obtain the binary probability distributions. Due to the linearity of our mixing model and the segment-wise structure, we can simplify the encoder (posterior approximation) of the VAE, and make all the transformations in the iVAE affine or linear, thus greatly simplifying the system. The linear iVAE is presented in more detail in Appendix E.

Estimation of the Sources
After estimating the mixing matrix A, it may be desired to estimate the sources z as well. We note that in the case of binary data, the individual source values cannot be accurately estimated (even up to scale and order indeterminacies) due to the inherent noise introduced by the binarization procedure. Presumably, though, if the number of observed variables is large and the number of sources is small, the estimation may be reasonable. In any case, the posterior p(z|x, u) can be easily calculated after estimating the mixing matrix.

EXPERIMENTS
We implemented our proposed methods and baselines using R (BLICA, full MLE) and python (linear iVAE). The methods will be published online in open source. Here we investigate empirically the identifiability of the model, as well as the finite-sample estimation performance and the scalability of our proposed methods, also comparing to previous approaches.
Data. Data was generated in the following way. Means were drawn from unif(−0.5, 0.5), standard devi-  ations from unif(0.5, 3). Mixing matrix elements were from unif(−3, 3) while ensuring invertibility by resampling until the condition number (kappa) was below 20 for n < 20, or for n ≤ 20 below the 75th quantile of 1000 sampled similar dimensional mixing matrices. For practical estimations from finite sample data we use 40 segments with varying the sample size.
Evaluation. ICA methods are often compared in terms of mean correlation coefficient of the estimated sources. Here however, binarization induces heavy noise and individual samples of the estimated sources cannot be accurately estimated. We therefore focus our evaluation on the mixing matrices, and measure the mean cosine similarity (MCS) of the columns (see Appendix D).
Identifiability: Results. We start by evaluating identifiability empirically. Crucially, since BLICA only employs pairwise marginals, we can use the exact binary distributions implied by the model as inputs, thus avoiding any effects of finite samples. Figure 3 shows results on which models can be identified when the number of sources equals the number of observations (n = n z ). In many cases, the method found the mixing matrix essentially up to machine precision, which can be seen as indication of identifiability. Each box includes 30 different data generating models, for each we ran BLICA 3 times; MCS of the run with highest scaled Gaussian likelihood is plotted. With only 2 segments, or only 2 variables, the model is not identifiable for any setting. Generally, the more observed variables we have, the less segments are needed for identification. The implications of the empirical results in Figure 3 are similar to the equations vs. unknowns arithmetic in Table 1. For example, 3 segments cannot identify models less than 9 variables.
Finite-sample estimation: Methods. Next we turn our attention to estimation performance from finite sample data. We compare our new BLICA (with different regularization parameter value r) method to its main competitors, fastICA (Himberg and Hyvärinen, 2001;Hyvärinen, 1999) and the baseline implementations of linear iVAE and full MLE. We note that the model of fastICA is somewhat different, but it still employs a linear mixing of the sources and has the same sources scale and order indeterminacies; thus, MCS comparison is sensible. fastICA does not use the segment index, but pools all data from different segments. Recall from Section E that the linear iVAE uses the same model, but instead of employing the likelihood, it optimizes the ELBO objective through L-BFGS. For runs with n < 20 observed variables a time budget of 2h was used, the results that were obtained within the time limit are reported. For larger simulations we allowed 12h per run. To avoid local minima due to the difficult optimization landscape, we ran linear iVAE, full MLE and BLICA with 3 different learning seeds and selected the best run according to the objective function (e.g. likelihood).
Finite-sample estimation: Results. Figure 4 (left) shows the result for 10 observed variables and 10 sources. BLICA clearly outperforms others consistently improving with increasing sample size. With smaller dimensions, 6 observed variables and 6 sources in Fig. 4 (center), more samples are needed. full MLE cannot perform sufficiently many optimization steps within the time limit of 2h. However, if the number of sources is limited we again achieve good estimation results: Fig. 4 (right) shows that for 6 observed variables and 2 sources, high MCS can be obtained with only 50 samples per segments. Interestingly, linear iVAE performs well only with fewer sources than observations, while fastICA is not able to reliably estimate the mixing matrix from binary data.
Scalability: Results. Figure 5 assesses the performance in higher dimensions over data sets with 40 1000-sample segments, thirty for each n. Only BLICA can estimate the mixing matrix with equal number of

RELATED WORK
Himberg and Hyvärinen (2001) consider binary observed vectors x and binary sources z, so that the ICA mixing model is given by the Boolean expression x i = nz j=1 a ij ∧ z j . They show that this Boolean OR mixing can be approximated by a linear mixing model followed by a unit step function. Thus, they propose to estimate the model by ordinary ICA, and obtain reasonable results when the data is very sparse. Similarly, Nguyen and Zheng (2011) studied binary ICA with OR mixtures by defining a disjunctive generative model. They prove identifiability and propose an algorithm without continuous-valued approximations. Kabán and Bingham (2006) proposed a model where continuous sources follow a Beta distribution, followed by a binary observation model. While their approach is related to ours, their latent variables are restricted to a finite interval, and they estimate the model using variational approximation which is unlikely to yield consistent estimators. Discrete ICA has further been approached by extensions of LDA where the topic intensities are mutually independent (Podosinnikova et al., 2015;Buntine and Jakulin, 2005;Canny, 2004). Although their identifiability guarantees are limited (Podosinnikova et al., 2016), their method has the ad-vantage of allowing for discrete, non-binary data. Lee and Sompolinsky (1999) consider PCA for binary data, employing a binarized Gaussian model.
Finally, we note that the very idea of estimating latent variable models by non-stationarity, originating in (Matsuoka et al., 1995;Pham and Cardoso, 2001), has been recently increasingly used in estimating generative models (Hyvärinen and Morioka, 2016; Khemakhem et al., 2020) as well as for causal discovery (Zhang et al., 2017;Monti et al., 2019), even in deep learning. Instead of the wide-spread idea of joint diagonalization of covariance matrices (Belouchrani et al., 1997;Tsatsanis and Kweon, 1998), we used correlation matrices without explicit diagonalization criteria; related work on diagonalizing correlation matrices can be found in (Joho and Rahbar, 2002).

CONCLUSION
We presented a model for ICA of binary data which is based on a linear latent mixing model and nonstationarity of the sources. We investigated the identifiability, showing some surprising indeterminacies not present in ordinary ICA, including the fact that in the two-variable case the model cannot be identified. We believe that our identifiability results, theoretical and empirical, will be useful in future research on binary ICA. Based on our approach using a Gaussian link function, the likelihood can be obtained in closed form although the Gaussian CDF is still computationally heavy. These advances allowed for a practical method BLICA, that combines maximum likelihood estimation and moment-matching; it was shown to be applicable in higher dimensions while still empirically showing consistent behaviour. As future work, we aim to also investigate which prospects the new algorithm opens in applications. A Proof of Row Order Indeterminacy for n = 2 (Theorem 1) Theorem 4. If the row order of the mixing matrix A of a two-dimensional binary ICA model is reversed, then the source means µ u z and variances Σ u z can be adjusted such that the implied binary distribution remains identical.

References
Proof. Consider two binary ICA models z } u and scaling matrices {Q u } u such that Equations 9 and 10 in the main paper are satisfied and therefore the binary distributions implied by both models for each segment are identical. First, let Σ u z = Σ u z . This and the row switching of A means that the covariance matrix of q has just the order switched: 1,2]. The equations implied by Equation 8 in the main paper for each u are: These can be solved by setting Finally, solve for µ u q from Equation 9 since A, A , Q u are invertible.

B Proof of Identifiability q-Correlations (Theorem 2)
Theorem 5. Two binary ICA models imply different binary distributions (in a given segment) if the correlation matrices for q are not equal.
We will first present the result assuming zero means for q since it is more approachable to the reader. Figure 1 explains this case visually. The full technical proof is given afterwards. Figures 2 and 3 explain the general case visually.
Proof assuming zero means. We can focus here on bivariate models as the multivariate normal for q can be straightforwardly marginalized to the bivariate case. Suppose the two models respectively imply: Due to Equations 9 and 10 in the main paper we can also assume we are dealing with "standardized" models where the diagonals of the covariances are units for both models. We get: The correlation/covariance matrices for q and q are: We study the difference in the implied binary distribution by the two models by creating the Gaussian distributions for q and q from a single standard multivariate Gaussian source. The distributions can be formed from a standard normal n ∼ N (0, I), for example by multiplying with matrices  5 (center). For this case with zero means, with higher correlation value α we get more 00 and 11 assignments as can be seen from the rightmost plot. Grey points in the rightmost plot do not imply extra 00 or 11 assignments with either correlation value and are irrelevant for the proof.
We will assume α > β without loss of generality. Let's look at which values for n result in different assignments for the binary variables. Recall that the assignment is determined deterministically by the quadrant q and q land in. Intuitively, the model with higher correlation α implies more similar values for the binary variables. For the α model: And for the β model: Note that due to the construction both models agree on the value of the binary variable x 1 .
With β we get extra assignments such that x 1 = x 2 = 0 if: Since α > β and x/ √ 1 − x 2 is increasing, the interval for n 2 is empty, and no n implies x 1 = x 2 = 0 with β if not with α. Suppose n is such that The binary values implied are x 1 = x 2 = 0 with α and x 1 = 0, x 2 = 1 with β. Since α > β and x/ √ 1 − x 2 is increasing, the interval for n 2 has non-zero measure. Thus there is a nonzero measure for obtaining extra x 1 = x 2 = 0 with α. See Figure 6 for pictorial representation of the situation when α = 0.5, β = −0.5.
Proof. We can focus here on bivariate models as the multivariate normal for q can be straightforwardly marginalized to the bivariate case. Suppose the two models respectively imply: Then the marginals are: P (x 1 = 1) = Φ(0|µ 1 , σ 2 1 ) = Φ(− µ 1 σ 1 |0, 1), where µ 1 , µ 1 , σ 1 , and σ 1 denote the parameters in Equation 13. For the models to imply the same distributions the marginals need to be the same. The same applies for x 2 with parameters µ 2 , µ 2 , σ 2 , and σ 2 . Since Φ is monotonically increasing, we can assume from here on: Due to Equations 9 and 10 in the main paper we can also assume we are dealing with "standardized" models where the diagonals of the covariances are units for both models. We get: The correlation/covariance matrices for q and q are: We study the difference in the implied binary distribution by the two models by creating the Gaussian distributions for q and q from a single standard multivariate Gaussian source. The distributions can be formed from a standard normal n ∼ N (0, I), for example by multiplying with matrices where µ = µ due to the earlier. We will assume α > β without loss of generality. Let's look at which values for n result in different assignments for the binary variables. Recall that the assignment is determined deterministically by the quadrant q and q land in. Intuitively, the model with higher correlation α implies more similar values for the binary variables. For the α model: And for the β model: Due to the construction both models agree on the value of the binary variable x 1 .
Case α > 0 With β we get additional assignments such that x 1 = x 2 = 0 if: Replacing n 1 with smaller µ 1 in the lower bound gives a necessary condition for this is: With β we get addditional assignments x 1 = x 2 = 1 if: Replacing n 1 with larger µ 1 in the upper bound gives a neccessary condition:  . For this case with µ 1 = −1, µ 2 = −1, with higher correlation value α we (provably) get more 00 assignments as can be seen from the rightmost plot. Grey points in the rightmost plot do not imply extra 00 or 11 assignments with either correlation value and are irrelevant for the proof.
Case α < 0 Also β < 0 here. With β we get additional assignments such that x 1 = x 2 = 0 if: Replacing βn 1 with larger βµ 1 in the upper bound gives a necessary condition for this is: With β we get additional assignments x 1 = x 2 = 1 if: Replacing βn 1 with smaller βµ 1 in the lower bound gives a neccessary condition: Since the upper bound of Equation 19 matches the lower bound of Equation 21, and the bound is constant with respect to n, both necessary conditions cannot be fulfilled given any fixed model. Therefore the conditions the previous were respectively necessary to, Equation 18 and Equation 20, will not be satisfied either for any fixed model. Note that either Equation 18 or Equation 20 can be satisfied alone.
The binary values implied are x 1 = x 2 = 0 with α and x 1 = 0, x 2 = 1 with β. Furthermore, the following shows that interval for −n 2 has non-zero measure. The first multiplication is permitted as the x/ √ 1 − x 2 is increasing . For this case with µ 1 = 1, µ 2 = 1, with higher correlation value α we (provably) get more 11 assignments as can be seen from the rightmost plot. Grey points in the rightmost plot do not imply extra 00 or 11 assignments with either correlation value and are irrelevant for the proof.
The binary values implied are x 1 = x 2 = 1 with α and x 1 = 1, x 2 = 0 with β. Furthermore, the folllowing shows that interval for −n 2 has non-zero measure. The first multiplication is permitted as the x/ √ 1 − x 2 is increasing