{'fraction_non_alphanumeric': 0.026488804192472605, 'fraction_numerical': 0.009306018739082102, 'mean_word_length': 5.060827718960539, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Mesoscopic Structure of the Stock Market and Portfolio Optimization

The idiosyncratic (microscopic) and systemic (macroscopic) components of market structure have been shown to be responsible for the departure of the optimal mean-variance allocation from the heuristic `equally-weighted' portfolio. In this paper, we exploit clustering techniques derived from Random Matrix Theory (RMT) to study a third, intermediate (mesoscopic) market structure that turns out to be the most stable over time and provides important practical insights from a portfolio management perspective. First, we illustrate the benefits, in terms of predicted and realized risk profiles, of constructing portfolios by filtering out both random and systemic co-movements from the correlation matrix. Second, we redefine the portfolio optimization problem in terms of stock clusters that emerge after filtering. Finally, we propose a new wealth allocation scheme that attaches equal importance to stocks belonging to the same community and show that it further increases the reliability of the constructed portfolios. Results are robust across different time spans, cross-sectional dimensions and set of constraints defining the optimization problem


Introduction
The pioneering work of Markowitz (1952) laid the foundations of modern portfolio theory through the mean-variance (MV) optimization procedure. According to that model, the portfolio optimizer deals with uncertainty either by minimizing the variance of the investment, given the expected return, or by maximizing the expected return, given a certain level of risk. Despite its simplicity, it is widely recognized that the mean-variance framework delivers a poor out-of-sample performance (Michaud, 1989;Bai et al., 2009). The MV predictions, in fact, seriously depart from empirical observations, thus questioning the need of individuating procedures for investment optimizationespecially when considering that the simpler 1/N heuristic rule, being less affected by covariance estimation errors, achieves better out-of-sample performances (Duchin and Levy, 2009). Indeed, as Laloux et al. (1999) show, the MV optimization procedure provides a biased estimation of the correlation matrix since the smallest eigenvalues of the latter, which play a fundamental role in the estimation of the global minimum variance (GMV) portfolio, are largely affected by noise.
In order to overcome the limitations of the MV framework, techniques have been proposed either to ameliorate its theoretical predictions or to capture the 'real' essence of the correlation matrix by means of which optimal portfolios are constructed through filtering procedures 1 . Overall, each different estimator and filtering procedure improves upon different portfolio aspects related to the performance, realized risk, reliability and diversification and these improvements also depend on other circumstances as the dimension-to-sample size ratio or the possibility of exploiting short-selling strategies.
The reader interested in the proof is redirected to Bai (1999). The result above implies that any empirical correlation matrix C of financial returns 6 , where the largest empirical eigenvalue is denoted as λ m and is usually (much) larger than λ max , can be decomposed as where |v i and v i | denote the column and row eigenvectors associated with the eigenvalue λ i respectively. The above decomposition represents the empirical correlation matrix as a sum of matrices respectively induced by the random spectral component C (r) , whose eigenvalues lie in the Marčenko-Pastur range [λ min , λ max ] and usually also below λ min (as a result of the fact that, since the trace of a correlation matrix should remain equal to N , the presence of λ m > λ max shifts the lower eigenvalues leftwards) and the structural (non-random) component C (s) . The filtering procedure consists in removing C (r) , i.e. the random component of the tensor: what remains is, then, recognized as signal rather than noise, hence supposed to possess useful economic information.
As mentioned above, the spectrum of empirical correlation matrices of financial returns is characterized by a leading eigenvalue λ m which is much larger than the others. The associated (column) eigenvector |v m possesses elements having the same sign and identifies a matrix component λ m |v m v m | affecting all stocks in the same direction and with strong intensity, further inducing the decomposition of C (s) as follows: i.e. as a sum of a mesoscopic spectral component C (g) and a systemic component (or market mode) C (m) . A graphical illustration of this empirical feature is provided in Figure 1 for the S&P500 constituents over the period 2000-2015 which constitute the dataset used in the remainder of the work as well. The systemic component is pervasive and time-varying; hence, some stocks might appear interconnected only as a consequence of their common dependence on global market events (see Forbes and Rigobon, 2002;Billio et al., 2012, among others). When performing asset allocation strategies based on historical data, it is fundamental to minimize covariance estimation errors induced by any possible time-varying component -which makes historical data not reliable for the future.
To provide an example, let us define the total risk of a system as Λ := N k=1 λ k and investigate the temporal evolution of the cumulative risk fraction of the different components of the covariance matrix of stock returns by adopting non-overlapping, rolling windows of two years. To this aim, can draw 100 randomized samples of size 100 from the S&P500 constituents, for each temporal window: the resulting averaged shares of total risk accounted by the random, systemic and mesoscopic component of the spectrum of the covariance matrix are shown in Figure  2. While the random and systemic cumulative risk fractions vary quite a lot across the considered period, the mesoscopic one is the most stable, a result letting us to conclude that the construction of more reliable portfolios may indeed be based on the stable part of the spectrum.
Let us now use C to partition the stock market into non-overlapping communities of stocks that are more correlated internally than expected under a suitable null model. Detecting communities in financial markets is not new in the literature: for instance, Fenn et al. (2012) compared different procedures to unfold the community structure of the foreign exchange market and Verma et al. (2019) used clusters to extract relevant factors for volatility modeling. However, the procedure we are now going to illustrate is based on a combination of modularity maximization (Clauset et al., 2004;Newman, 2006) and RMT (MacMahon and Garlaschelli, 2015), which was shown to be theoretically superior in the case of correlation matrices.
In the network science literature, the so-called modularity Q(γ) of a partition γ of the N nodes of a network is defined as where w ij the entry of the adjacency matrix of the (possibly weighted) network (i.e. w ij is the weight of the link from node i to node j), w ij is its expected value under a suitably chosen null model, and the Kronecker delta δ(γ i , γ j ) guarantees that only the nodes belonging to the same community contribute to the modularity. The goal of modularity maximization is finding the partition that maximizes Q(γ), thus emphasizing the community of nodes whose internal interactions are stronger and maximally unexplained by the (community-free) null model. For networks, the null model chosen is generally the so-called Weighted Configuration Model (WCM) that randomizes the network topology while preserving the empirical strength s i = N j=1 w ij of each node i. A popular, although in general incorrect (Garlaschelli and Loffredo, 2009), expression used to represent this null model is N j=1 w ij is the total edge weight of the network. When considering correlation matrices, the null model above has been shown to be inconsistent (MacMahon and Garlaschelli, 2015) as a result of the fact that, unlike (weighted) networks, correlation matrices cannot be directly randomized by considering their entries as independent. Rather, the randomization should occur at the level of the underlying time series, and the correlation matrix should then be recalculated from the randomized time series. In particular, by reformulating the modularity for correlation matrices as a consistent community-free null model representing random empirical correlations resulting only from noise and possibly global trends comes precisely from RMT and can be expressed as (MacMahon and Garlaschelli, 2015). The above null model discounts both the random and the systemic components of correlations. As a consequence, i.e. the modularity matrix coincides with the mesoscopic component of the original correlation matrix. Therefore maximizing the modularity Q(γ) guarantees that the identified communities are necessarily formed by internally positively (after discounting the null model) and mutually negatively (again, after discounting the null model) correlated stocks. In other words, the communities are ideally noise-free and mutually anti-correlated with respect to the market.

Stock market communities
The dataset used for the present analysis has been downloaded from Yahoo Finance and consists of equity data for the 450 most capitalized companies in the US, stably traded over the last 20 years, all constituting the S&P500. After applying RMT to isolate the mesoscopic component of the matrix, we performed the modularity maximization procedure by implementing a modified version of the Louvain algorithm (Blondel et al., 2008), taking as input the matrix C (g) . The consistency and stability of this approach have been discussed in MacMahon and Garlaschelli (2015) and Anagnostou et al. (2021) to which the interested reader is referred for additional technical clarifications. We conducted the above analysis across the time span 2000-2015 and identified an optimal partition of the 450 stocks into 4 communities. Figure 3 shows the heatmaps depicting the sequence of transformations leading from the original stocks to such a set of mutually, negatively correlated communities. Figure 4 shows the detected communities, together their relative compositions, according to the industrial classification. The number of detected communities is lower than the number of considered sectors, showing the tendency of stocks to be strongly interconnected across different sectors as well. Still, it can be noticed how stocks belonging to specific sectors tend to cluster more than others -a behavior detected also in Borghesi et al. (2007) by employing hierarchical clustering techniques. In particular, almost all stocks in the financial sector are clustered together in C3 while stocks in the energetic and technological sectors are respectively placed in C4 and C1; utilities are quite clustered in C2. The remaining sectors (namely the industrials, materials, consumer discretionary, consumer staples and healthcare), instead, are more dispersed across different communities. This result confirms that the data-driven cluster identification leads to communities that are unpredictable from the nominal sectoral classification of stocks, as also observed in MacMahon and Garlaschelli (2015) and Anagnostou et al. (2021).
Performing the community detection over the whole available time span, i.e. from 2000 to 2015, might seem unreasonable since structural changes have arguably occurred in occasion of the 2008 financial crisis and possibly other events. This is only partially true: it turns out that while the original, unfiltered empirical correlations do change a lot over time (especially during market turmoils), mesoscopic correlations remain remarkably stable, in turn stabilizing the optimal partition. This can be easily seen by comparing the evolution of the density of the unfiltered, empirical correlations of our sample of stocks with that of the filtered, mesoscopic correlations employed to perform the clustering procedure. As Figure 5 shows, the distribution of the empirical correlation coefficients clearly shifts toward higher values in the second half of the considered time span (which contains a period of higher turmoil), so that the coefficients calculated over the entire time span are not representative of the underlying sub-periods. By contrast, when considering only the mesoscopic component of the correlation matrix over different periods, we find that the distribution of the entries of such component almost perfectly overlap with each other over time. In this case, the overall distribution is representative of the distributions for the sub-periods.
Since the stability of a distribution does not necessarily imply the stability of the individual entries of the matrix components, as a more stringent test we look at the evolution of the norm of the matrix containing the relative changes of each coefficient. Given the generic entry C The above quantity is the relative variation of the correlation coefficient across two consecutive time spans. We focus on relative variations since the components of the correlation matrix are characterized by magnitudes that are so different that a comparison of absolute changes would be meaningless. The stability of each component of the correlation matrix can be inspected by employing the p-norm ||∆C (f ) || p . As we can see from Table 1, the mesoscopic component C (g) turns out to be the most stable over time, given its smallest temporal variations.
To further assess the stability of the mesoscopic part of the correlation matrix, we repeat the computation just described by arranging the terms of C (g) according to the community a given node belongs to, so to have a matrix arranged in blocks, as displayed in Figure 3. Afterwards, to attach equal importance to nodes according to the community they belong to, we replace the values in each given block with the average computed over the block itself and denote this new matrix as C (g) . Notice that no clear difference arises with respect to the norms computed for C (g) , a result confirming the stability of the detected structures. Taken together, all the above checks of the stability of the filtered, mesoscopic component of empirical correlations lay the ground for our subsequent analyses in the rest of the paper.

Back to basic portfolio optimization
Let us now address the implications of the market mesoscopic structure from a portfolio management perspective. In order to do so, let us briefly review the classical Markowitz portfolio optimization scheme. Consider N risky assets with covariance matrix Σ and vector of expected returns µ. Given the wealth allocation vector ω = [ω 1 . . . ω N ], such that i ω i = 1, the portfolio expected return reads Figure 5: Densities of the unfiltered empirical (top) and filtered mesoscopic (bottom) correlation coefficients for S&P constituents over different periods. Notice that, while a clear shift occurs between the first and the second half of the overall 2000-2015 period for the unfiltered coefficients, no shift occurs for the filtered mesoscopic ones. As a consequence, the distribution of the unfiltered matrix entries calculated over the entire period is not representative of the distributions for the individual sub-periods, while that of the filtered matrix entries is.
with associated variance reading The well-known Markowitz approach consists in finding the allocation vector ω which minimizes σ 2 p subject to a given value of µ p or, equivalently, the one that maximize the return subject to a given level of variance. The optimization problem to be solved, expressed in matrix form, reads and has solution In this work, we focus on the variance and consider a completely risk-adverse investor that is only interested in minimizing the risk with no constraints on the expected return. In that case, the solution simply becomes where ω gmv denotes the investment plan associated with the global minimum variance (GMV) portfolio. We then focus on the reliability of an optimal portfolio by comparing its predicted risk, σ p , obtained via the correlation matrices estimated using historical data, with the (ex-post) realized risk, σ r p . As in Tola et al. (2008), we deem a portfolio as reliable if is 'small' -the main difference of our approach being that we will never assume perfect knowledge of future volatilities for the investor, letting uncertainty affect the whole covariance matrix.

Noise-free and systemic-free optimization
As shown before, the systemic component affects all stocks in the same direction, inducing a positive amount of covariance between the variables, i.e. σ (m) ij > 0, it is straightforward to show that, for a risk-minimizer investor, the adoption of the mesoscopic variance σ holds) rebalances the portfolio: hence, the total wealth will not be concentrated anymore over few assets characterized by the lowest past variances. In other words, while in presence of co-movements (e.g. because of market turmoils), an 'ingenuous' investor would (try to) lower the portfolio risk by concentrating the wealth over the less risky assets, an investor who is aware of the temporarily nature of aggregate shocks causing crashes in the market, would filter out the systemic effects from past data and trust only the stable part of the correlation matrix, for the future.
To provide empirical evidence for such a statement, we performed the MV optimization procedure on the S&P500 constituents over multiple periods, comparing the wealth allocation vectors obtained using the empirical and the mesoscopic correlation matrices and keeping the equally-weighted portfolio as a benchmark 7 : Figure 6 shows the results, considering both cases in which short selling is either possible or not. Noticeably, the MV optimization based on mesoscopic correlations closely follow the 1/N rule, yielding as an optimal solution a portfolio which is very similar to the equally-weighted one; on the contrary, the standard Markowitz optimization framework outputs a much more heterogeneous composition being more sensitive to the estimated sample covariances. This confirms what expected, i.e. that the optimization procedure based on the mesoscopic structure of the correlation matrix is less sensitive to both noisy and aggregated fluctuations, thus yielding more balanced portfolios. As an additional test, in Figure 7 we compare the mesoscopic and 1/N weights with the ones we would obtain by cleaning the correlation matrix only from noise through the standard RMT-based approach: in order to closely track the balanced 1/N allocation it is necessary to filter out both the noisy and the systemic components.
A measure of similarity to the equally-weighted portfolio is provided by the number of stocks with a 'significant' amount of money invested into. Following Bouchaud and Potters (2003), this quantity can be defined as indeed, when the wealth is equally divided among the N assets, the quantity N is equal to N ; on the other had, it is equal to 1 when the wealth is invested only in one asset. As stressed in Tola et al. (2008), the quantity N simply provides a rough estimate of the number of stocks which could be effectively used to build a portfolio that is smaller than the original but preserves most of the risk-return properties of the latter. In Figure 8, the effective size of the portfolios obtained with different allocation rules are displayed and compared: the compared allocation strategies are, again, the Markowitz GMV portfolios, the 1/N rule, the mesoscopic-based GMV portfolios and, finally, the noise-free GMV portfolios. It can be noticed that, independently from factors such as the time span, the subsample and the subsample size considered, the mesoscopic-based GMV portfolios are always much closer to the 1/N rule than those output by the classical Markowitz approach and the one based on RMT but filtering out only the random component. As it will be shown afterwards, this result brings non-trivial practical implications in terms of reliability.

Mesoscopic community-based optimization scheme
The adoption of the mesoscopic correlations leads to balanced portfolios closely tracking the equally-weighted investment plan. Let us now show how the portfolio optimization problem can be simply reformulated by taking into account the detected clusters of stocks, instead of the single ones, to further reduce the uncertainty characterizing each specific asset.
Let N = N 1 + N 2 + · · · + N n be the total number of asset, n the number of detected communities, N c being the number of assets in a given community (denoted by the subscript c ∈ {1, 2 . . . n}). The problem, now, is that of finding the share of wealth W c which has to be invested into a given community, with ω c = W c /N c being the share of wealth invested into the generic asset i belonging to that community. The problem can be, thus, reformulated as follows and is the same as the problem in 3 with the difference that weights are constrained to be equal for all the stocks belonging to the same community c; naturally, the total wealth share, being the sum of the wealth shares invested  Markowitz (red) and the portfolio optimization based on mesoscopic correlations (blue). For each stock on the x-axis, the relative weight on the y-axis is shown, the mesoscopic-based optimization closely follow the heuristic 1/N rule. When short-selling is not allowed we have ω i ≥ 0, ∀ i.   Table 2: Reliability R for each strategy adopted and for each sample size, under different time spans, ranging from the N = 50 case to the whole sample case (N = 450). No randomization occured when the whole sample is taken. The measures R mesoscopic and R Markowitx refer to the GMV portfolio cases with and without short-selling strategies. Entries in bold refer to the strategies with the best performances, both with and without short-selling constraints. Whenever the performance is worse than the equally weighted portfolio, the relative entry is not highlighted.
into each community, must still sum up to one. Reformulating the problem as in 16 leads to the minimization of the following objective function, where no constraint on the expected return is placed: notice that σ 2 c and σ ck respectively denote the average of the variances inside a given community and the average of the mesoscopic covariances between assets belonging to different communities. The reliability analysis of the proposed approach will be assessed in the next section.

Reliability analysis
Let us now compare the reliability R of the portfolios obtained by implementing the classical Markowitz portfolio optimization approach, the mesoscopic-based optimization approach, the mesoscopic plus community-based optimization approach and, finally, the heuristic equally-weighted strategy.
Both cases with and without short-selling will be analyzed, focusing on the GMV portfolios computed over different periods and for different sample sizes. Then, for the sake of completeness, we repeat the comparison considering 30 values of expected portfolio return µ p to insert, as additional constraints, on the minimization problem. The different portfolios constituting the efficient frontier are, then, computed, and for each of them the index R is obtained, given the out-of-sample realized portfolios variances. The analyses are carried out over different time-spans and considering different sample sizes: the time-spans analyzed, i.e. T 1 = 2000 − 2007, T 2 = 2004 − 2011 and T 3 = 2008 − 2015 are divided in two additional subperiods of equal length by fixing t 0 . Upon doing so, we create porfolios given the data collected over the period t 0 − ∆t and quantify their out-of-sample performance over the period t 0 + ∆t. For what concerns the samples size, we randomly extract 100 subsamples out of the S&P500 components, for each considered size.
Average values computed for the R indices are reported in Table 2. When short-selling is allowed and no constraint on the expected portfolio return is present, the Markowitz approach is always underperforming -the only exception being represented by the lowest-dimensionality case (N = 50) -when compared with the 1/N and the mesoscopic-based optimization rule, irrespectively from the sample size and the time span considered. In addition,  Table 3: Comparison between the community-based portfolios and the other methodologies. With R * mesoscopic and R * Markowitz we indicate that we chose the best performance between the shortselling and no short-selling cases our methodology performs slightly better than the equally-weighted portfolio, thus revealing it to be the most reliable investment plan considered: the difference, however, is almost negligible, a result confirming the closeness of the mesoscopic-based optimization procedure and the equally-weighted strategy. From a purely mathematical perspective, imposing constraints is equivalent at letting a shrinkage operator act on the covariance matrix of the assets, an operation helping when the number of parameters to estimate is too large -and, as a consequence, estimation errors are large as well.
The poor performance of the GMV Markowitz portfolios is not a novel result, especially when no constraint about the possibility of exploiting short-selling strategies is imposed (see Frost and Savarino, 1988;Eichhorn et al., 1998;Britten-Jones, 1999;Jagannathan and Ma, 2003). Our empirical analysis confirms these results: notice the huge improvement of the Markowitz approach compared to the situation in which short-selling was not allowedalthough the mesoscopic-based optimization provieds better reliability indices in all cases except in period T 3 .
Let us now check whether the clusters detected on C (g) can be used as a further source of information. In particular, let us attach homogeneous optimal weights to stocks belonging to the same clusters, denoting with R community the corresponding reliability index. To make the comparison as clear as possible, we compare the reliability of the community-based portfolios only with the best performing competing approach, in each time span. Results in Table 3 noticeably confirm the informativeness of the detected communities from a risk-management perspective. Portfolios in which optimal weights are recovered by constraining stocks in the same community to weigh the same further improve their reliability indices, outperforming both the equally-weighted and the mesoscopicbased strategies for all considered time spans. Still, Markowitz with the no short-selling constraint is the more reliable in T 3 .
Let us now consider all approaches, i.e. the classical Markowitz one, the mesoscopic-based one and the mesoscopic plus community-based one and compute the reliability index for each portfolio of each efficient frontier. Results are summarized in Table 4, where the R indices are ordered and compared between the different quartiles of the expected return distribution 8 : when adding constraints on expected returns, Markowitz outperforms our methodology, in time span T 2 , when also the constraint on short-selling is imposed and in time span T 3 after the first quartile of the distributions? In time span T 1 , instead, the community-based approach outperform the others. Overall, our approach is confirmed to perform better in all periods when we impose constraints only on expected returns but not on the weights. In particular, optimizing by taking into account the detected clusters stabilize the results, hence providing the best reliability.
Providing a deep explanation for such a result is hard given the higher degree of uncertainty introduced by the constraints on the expected returns. What is clear, however, is that cleaning the correlation matrices from both noise and systemic effects helps to ameliorate the reliability of the minimum variance portfolios and exploiting stocks communities identified through the mesoscopic correlation further improves the results. The same holds true when constraints on expected returns are imposed but allowing for short-selling strategies. When both constraints on returns and weights are in place, however, Markowitz approach is found to be hardly beatable.

Discussion and conclusions
In this work we investigated the mesoscopic structure of the stock market correlations that emerge after filtering out both microscopic (stock-specific noise) and macroscopic (market-wide trends) components. We showed that such mesoscopic correlations are the most stable over time, thereby encoding important information in the context of portfolio optimization. Indeed, we found that the noisy and the systemic components of the stock market are unstable, leading to biased and poor out-of-sample performances and being responsible for the surprising departure of the classical Markowitz investment prescription from the heuristic, equally weighted strategy. Upon filtering out these unstable components, the market can be partitioned into internally positively and mutually negatively correlated communities of stocks. We proposed to use these stable mesoscopic communities to construct portfolios characterized by higher levels of reliability in terms of predicted and realized risk. Results can be summarized as follows. The adoption of 'noise-and systemic-free' correlations leads to an asset allocation which closely tracks, and slightly outperforms, the reliability of the heuristic equally weighted portfolio, while at the same time requiring a smaller number of assets over which the wealth need be effectively invested. In addition, both the equally weighted portfolios and the ones induced by the proposed optimization scheme have been found to be more reliable than the Markowitz plug-in estimator. Importantly, the reliability of portfolios can be further improved by performing the mesoscopic optimization while simultaneously accounting for the community to which a given stock belongs: remarkably, also when constraints on short-selling are imposed, this new methodology performs noticeably better than classical Markowitz.
Only when constraints on both weights and expected returns are imposed, the homogeneous community-based portfolios do not bring improvements compared to classical Markowitz -with the exception of the period T 1 = 2000 − 2007 and for few specific levels of targeted expected returns. Thus, the proposed methodology works well when focusing on the minimum-variance portfolio or when short-selling can be performed, suggesting the adoption of network clustering techniques for risk management applications. In particular, the uncovered mesoscale structure might bring insights about additional, and complementary, ways of creating stock market indices to monitor market trends -something which might be the object of further studies aimed at understanding co-movements between industries and sectors in the stock market. systemic effects. Maximizing with respect to the n detected communities, so to have homogeneous weights inside a given community, can be achieved by splitting the variance of the portfolio as follows Nn j=1 ω 2 ω n σ ij2n . . .
Nn j=1 ω n−1 ω n σ ij(n−1)n which is equivalent to Thus the objective function to minimize with respect to the community weights become with W c = ω c N c being the total share of wealth invested in community c.