# Noise Dressing of Financial Correlation Matrices

\[

## Abstract

We show that results from the theory of random matrices are potentially of great interest to understand the statistical structure of the empirical correlation matrices appearing in the study of price fluctuations. The central result of the present study is the remarkable agreement between the theoretical prediction (based on the assumption that the correlation matrix is random) and empirical data concerning the density of eigenvalues associated to the time series of the different stocks of the S&P500 (or other major markets). In particular the present study raises serious doubts on the blind use of empirical correlation matrices for risk management.

\] An important aspect of risk management is the estimation of the correlations between the price movements of different assets. The probability of large losses for a certain portfolio or option book is dominated by correlated moves of its different constituents – for example, a position which is simultaneously long in stocks and short in bonds will be risky because stocks and bonds move in opposite directions in crisis periods. The study of correlation (or covariance) matrices thus has a long history in finance, and is one of the cornerstone of Markowitz’s theory of optimal portfolios . However, a reliable empirical determination of a correlation matrix turns out to be difficult: if one considers $`N`$ assets, the correlation matrix contains $`N(N1)/2`$ entries, which must be determined from $`N`$ time series of length $`T`$; if $`T`$ is not very large compared to $`N`$, one should expect that the determination of the covariances is noisy, and therefore that the empirical correlation matrix is to a large extent random, i.e. the structure of the matrix is dominated by measurement noise. If this is the case, one should be very careful when using this correlation matrix in applications. In particular, as we shall show below, the smallest eigenvalues of this matrix are the most sensitive to this ‘noise’ – on the other hand, it is precisely the eigenvectors corresponding to these smallest eigenvalues which determine, in Markowitz theory, the least risky portfolios . It is thus important to devise methods which allows one to distinguish ‘signal’ from ‘noise’, i.e. eigenvectors and eigenvalues of the correlation matrix containing real information (which one would like to include for risk control), from those which are devoid of any useful information, and, as such, unstable in time. From this point of view, it is interesting to compare the properties of an empirical correlation matrix C to a ‘null hypothesis’ purely random matrix as one could obtain from a finite time series of strictly uncorrelated assets. Deviations from the random matrix case might then suggest the presence of true information. The theory of Random Matrices has a long history in physics since the fifties , and many results are known . As shown below, these results are also of genuine interest in a financial context (see also ).

The empirical correlation matrix C is constructed from the time series of price changes<sup>*</sup><sup>*</sup>*In the following we assume that the average value of the $`\delta x`$’s has been subtracted off, and that the $`\delta x`$’s are rescaled to have a constant unit volatility. $`\delta x_i(t)`$ (where $`i`$ labels the asset and $`t`$ the time) through the equation:

$$\text{C}_{ij}=\frac{1}{T}\underset{t=1}{\overset{T}{}}\delta x_i(t)\delta x_j(t).$$
(1)   
We can symbolically write Eq. (1) as C$`=`$ 1/T M M$`^\text{T}`$, where M is a $`N\times T`$ rectangular matrix, and $`^\text{T}`$ denotes matrix transposition. The null hypothesis of uncorrelated assets, which we consider now, translates itself in the assumption that the coefficients $`M_{it}=\delta x_i(t)`$ are independent, identically distributed, random variablesNote that even if the ‘true’ correlation matrix C$`_{\text{true}}`$ is the identity matrix, its empirical determination from a finite time series will generate non trivial eigenvectors and eigenvalues.. We will note $`\rho _C(\lambda )`$ the density of eigenvalues of C, defined as:

$$\rho _C(\lambda )=\frac{1}{N}\frac{dn(\lambda )}{d\lambda },$$
(2)   
where $`n(\lambda )`$ is the number of eigenvalues of C less than $`\lambda `$. Interestingly, if M is a $`T\times N`$ random matrix, $`\rho _C(\lambda )`$ is exactly known in the limit $`N\mathrm{}`$, $`T\mathrm{}`$ and $`Q=T/N1`$ fixed , and reads:

$`\rho _C(\lambda )`$ $`=`$ $`{\displaystyle \frac{Q}{2\pi \sigma ^2}}{\displaystyle \frac{\sqrt{(\lambda _{\text{max}}\lambda )(\lambda \lambda _{\text{min}})}}{\lambda }},`$ (3)   
$`\lambda _{\text{min}}^{\text{max}}`$ $`=`$ $`\sigma ^2(1+1/Q\pm 2\sqrt{1/Q}),`$ (4)   
with $`\lambda [\lambda _{\text{min}},\lambda _{\text{max}}]`$, and where $`\sigma ^2`$ is equal to the variance of the elements of M , equal to $`1`$ with our normalisation. In the limit $`Q=1`$ the normalised eigenvalue density of the matrix $`𝐌`$ is the well known Wigner semi-circle law, and the corresponding distribution of the square of these eigenvalues (that is, the eigenvalues of C) is then indeed given by (4) for $`Q=1`$. The most important features predicted by Eq. (4) are:

* the fact that the lower ‘edge’ of the spectrum is strictly positive (except for $`Q=1`$); there is therefore no eigenvalues between $`0`$ and $`\lambda _{\text{min}}`$. Near this edge, the density of eigenvalues exhibits a sharp maximum, except in the limit $`Q=1`$ ($`\lambda _{\text{min}}=0`$) where it diverges as $`1/\sqrt{\lambda }`$.
* the density of eigenvalues also vanishes above a certain upper edge $`\lambda _{\text{max}}`$.

Note that the above results are only valid in the limit $`N\mathrm{}`$. For finite $`N`$, the singularities present at both edges are smoothed: the edges become somewhat blurred, with a small probability of finding eigenvalues above $`\lambda _{\text{max}}`$ and below $`\lambda _{\text{min}}`$, which goes to zero when $`N`$ becomes large.

Now, we want to compare the empirical distribution of the eigenvalues of the correlation matrix of stocks corresponding to different markets with the theoretical prediction given by Eq. (4), based on the assumption that the correlation matrix is random. We have studied numerically the density of eigenvalues of the correlation matrix of $`N=406`$ assets of the S&P 500, based on daily variations during the years 1991-96, for a total of $`T=1309`$ days (the corresponding value of $`Q`$ is $`3.22`$).

An immediate observation is that the highest eigenvalue $`\lambda _1`$ is $`25`$ times larger than the predicted $`\lambda _{\text{max}}`$ – see Fig. 1, inset. (The corresponding eigenvector is, as expected, the ‘market’ itself, i.e. it has roughly equal components on all the $`N`$ stocks.) The simplest ‘pure noise’ hypothesis is therefore inconsistent with the value of $`\lambda _1`$. A more reasonable idea is that the components of the correlation matrix which are orthogonal to the ‘market’ is pure noise. This amounts to subtracting the contribution of $`\lambda _{\text{max}}`$ from the nominal value $`\sigma ^2=1`$, leading to $`\sigma ^2=1\lambda _{\text{max}}/N=0.85`$. The corresponding fit of the empirical distribution is shown as a dotted line in Fig. 1. Several eigenvalues are still above $`\lambda _{\text{max}}`$ and might contain some information, thereby reducing the variance of the effectively random part of the correlation matrix. One can therefore treat $`\sigma ^2`$ as an adjustable parameter. The best fit is obtained for $`\sigma ^2=0.74`$, and corresponds to the dark line in Fig. 1, which accounts quite satisfactorily for 94% of the spectrum, while the 6% highest eigenvalues still exceed the theoretical upper edge by a substantial amount. Note that still a better fit could be obtained by allowing for a slightly smaller effective value of $`Q`$, which could account for the existence of volatility correlations .

We have repeated the above analysis on different stock markets (e.g. Paris) and found very similar results. In a first approximation, the location of the theoretical edge, determined by fitting the part of the density which contains most of the eigenvalues, allows one to distinguish ‘information’ from ‘noise’. However, a more careful study should be undertaken, in particular to treat adequately the finite $`N`$ effects.

The idea that the low lying eigenvalues are essentially random can also be tested by studying the statistical structure of the corresponding eigenvectors. The $`i^{th}`$ component of the eigenvector corresponding to the eigenvalue $`\lambda _\alpha `$ will be denoted as $`v_{\alpha ,i}`$. We can normalise it such that $`_{i=1}^Nv_{\alpha ,i}^2=N`$. If there is no information contained in the eigenvector $`v_{\alpha ,i}`$, one expects that for a fixed $`\alpha `$, the distribution of $`u=v_{\alpha ,i}`$ (as $`i`$ is varied) is a maximum entropy distribution, such that $`\overline{u^2}=1`$. This leads to the so-called Porter-Thomas distribution in the theory of random matrices:

$$P(u)=\frac{1}{\sqrt{2\pi }}\mathrm{exp}\frac{u^2}{2}.$$
(5)   
As shown in Fig. 2, this distribution fits extremely well the empirical histogram of the eigenvector components, except for those corresponding to the highest eigenvalues, which lie beyond the theoretical edge $`\lambda _{\text{max}}`$. We show in the inset the distribution of $`u`$’s for the highest eigenvalue, which is markedly different from the ‘no information’ assumption, Eq. (5).

We have finally studied correlation matrices corresponding not to price variations but to the (time dependent) volatilities of the different stocks, determined from the study of intraday fluctuations. These matrices should contain some relevant information for option trading and hedging. The obtained results are again very similar to those shown in Fig. 1 and 2.

To summarise, we have shown that results from the theory of random matrices (well documented in the physics literature ) is of great interest to understand the statistical structure of the empirical correlation matrices. The central result of the present study is the remarkable agreement between the theoretical prediction and empirical data concerning both the density of eigenvalues and the structure of eigenvectors of the empirical correlation matrices corresponding to several major stock markets. Indeed, in the case of the S&P 500, 94% of the total number of eigenvalues fall in the region where the theoretical formula (4) applies. Hence, less than 6% of the eigenvectors which are responsible of 26% of the total volatility, appear to carry some information. This method might be very useful to extract the relevant correlations between financial assets of various types, with interesting potential applications to risk management and portfolio optimisation. It is clear from the present study that Markowitz’s portfolio optimisation scheme based on a purely historical determination of the correlation matrix is not adequate, since its lowest eigenvalues (corresponding to the smallest risk portfolios) are dominated by noise.

Acknowledgements: We want to thank J.P. Aguilar (CFM), J.M. Lasry (Paribas), M. Meyer (Science & Finance) and S. Galluccio (Paribas) for discussions.