{'fraction_non_alphanumeric': 0.03558145776618995, 'fraction_numerical': 0.025344120240788314, 'mean_word_length': 4.415697674418604, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Quantifying cosmic variance

We determine an expression for the cosmic variance of any"normal"galaxy survey based on examination of M* +/- 1 mag galaxies in the SDSS DR7 data cube. We find that cosmic variance will depend on a number of factors principally: total survey volume, survey aspect ratio, and whether the area surveyed is contiguous or comprised of independent sight-lines. As a rule of thumb cosmic variance falls below 10% once a volume of 10^7h_0.7^-3Mpc^3 is surveyed for a single contiguous region with a 1:1 aspect ratio. Cosmic variance will be lower for higher aspect ratios and/or non-contiguous surveys. Extrapolating outside our test region we infer that cosmic variance in the entire SDSS DR7 main survey region is ~7% to z<0.1. The equation obtained from the SDSS DR7 region can be generalised to estimate the cosmic variance for any density measurement determined from normal galaxies (e.g., luminosity densities, stellar mass densities and cosmic star-formation rates) within the volume range 10^3 to 10^7 h^-3_0.7Mpc^3. We apply our equation to show that 2 sightlines are required to ensure cosmic variance is<10% in any ASKAP galaxy survey (divided into dz ~0.1 intervals, i.e., ~1 Gyr intervals for z<0.5). Likewise 10 MeerKAT sightlines will be required to meet the same conditions. GAMA, VVDS, and zCOSMOS all suffer less than 10% cosmic variance (~3%-8%) in dz intervals of 0.1, 0.25, and 0.5 respectively. Finally we show that cosmic variance is potentially at the 50-70% level, or greater, in the HST Ultra Deep Field depending on assumptions as to the evolution of clustering. 100 or 10 independent sightlines will be required to reduce cosmic variance to a manageable level (<10%) for HST ACS or HST WFC3 surveys respectively (in dz ~ 1 intervals). Cosmic variance is therefore a significant factor in the z>6 HST studies currently underway.


INTRODUCTION
The Universe is not homogeneous except on the largest scales (> 1Gpc, Davis et al. 1985). As a consequence number and density measurements derived from within modest volumes will show greater than Poisson variation (see Szapudi & Colombi 1996 for example). This cosmic variance 1 , or small-scale scale-dependent inhomogeneity, is often the dominant source of error in many contemporary extragalactic measurements. Examples include the galaxy luminosity Jackknife sampling of the volume in question (e.g., Liske et al. 2003). These methods all have strengths and weaknesses can be laborious to impliment and potentially inconsistent depending on the method adopted and the assumptions made. For example to estimate the cosmic variance from numerical simulations requires the adoption of a numerical simulation (e.g., Springer et al. 2005), and either a semi-analytical prescription (Cole et al. 2000;Baugh 2006), or a halo occupation distribution (Berlind & Weinberg 2002;Moster et al. 2010) before an appropriate cosmic variance estimate can be made (e.g., Moster et al. 2010). The analytical method (e.g., Driver et al. 2003) implicitly assumes Poisson statistics, radial symmetry, and (because it is parametric) smoothes over potential 'features' in the underlying distribution (e.g., BAOs). The empirical method is not always practical if a large suitable survey does not exist, and Jackknife sampling (where one divides the sample into many parts and recomputes the value in question with each part missing in turn) is only capable of revealing the cosmic variance on scales smaller than the volume in question (nevertheless a useful indicator as cosmic variance should generally decrease with increasing scale).
In this paper we aim to use the largest volume survey to date, the Sloan Digital Sky Survey (SDSS), to empirically determine some generically useful formulae for estimating the cosmic variance as a function of survey volume and survey shape. These formulae should also assist in the design of future surveys where trade offs between area and depth need to be made. Throughout we adopt a standard cosmology with the following parameter set: ΩM = 0.3, ΩΛ = 0.7, Ho = 70kms −1 Mpc −1 although as our analysis is based on very local volumes only the value of the adopted Hubble constant is significant.

THE CALIBRATION SAMPLE
The Sloan Digital Sky Survey Data Release 7 (SDSS DR7; Abazajian et al. 2009) represents the final release of the original main galaxy survey programme. As our starting point we downloaded the main galaxy survey spectroscopic compendium catalogs 2 . In total this comprises of the full DR1-7 spectroscopic catalogues plus files of extra regions (mainly re-observations of poor quality plates), and two files of special regions (mainly Stripe 82 observations and leading into SEGUE). From these data the following fits keyword columns were selected: ra, dec, zfinal, zconffinal, petrocounts, reddening. From this extraction only r-band fluxes were selected and these were corrected for reddening using the reddening values provided so that in what follows, ro = r petrocounts − r reddening in mag units. Fig. 1 (upper) shows the distribution of these data on the sky colour coded according to the SDSS data release evolution. Estimating the coverage of these data is non-trivial, a simple method of counting occupied cells of equal area across the sky suggests it is between 8200 and 8000 sq degrees. The main difficulty is the complex boundary shape. To simplify the calculations and minimise boundary concerns we extracted a well defined sub-region indicated on Fig. 1 (upper) as a black dashed line. This is specified by the coordinates 130.00 < α(J2000.0) < 236.00, 0.00 < δ(J2000.0) < 58.00 representing a pyramidic section of the sky. The area enclosed within this section is 5150 sq.deg. While holes within this data exist, mainly small regions due to bright stars, this is to some extent another kind of cosmic variance (foreground obscuration) operating on a scale significantly smaller than that in which we are interested (> 1 sq.deg). We therefore elect not to use the SDSS mask for four reasons: firstly speed of computation (in our analysis we will be extracting many millions of areas and correcting for holes will slow the code considerably), secondly the area loss is < 3%, thirdly it should not correlate with background structure, and fourthly and perhaps most importantly the foreground obscuration is a real phenomena.
In our selected sub-region we find 463k galaxies with ro < 17.77 mag of which 415k (90 per cent) have reliable redshifts (zconffinal> 0.95). To assess the uniformity of the data over the entire region we construct the integrated number-counts to ro < 17.77 mag in 0.1 sq deg cells (as shown in Fig. 1 lower) and perform a bilinear fit to a simple flat projection of this region (i.e., we interpret Right Ascension and Declination as Cartesian but keep cell sizes as strictly 0.32 × 15 cos(δ)0.32 to generate uniform 0.1 sq deg cells). The fit reveals a weak gradient in both Right Ascension and Declination, caused by the presence of a series of superclusters in the lower right corner of Fig. 1 (lower). This illustrates that even the SDSS is not immune to cosmic variance exhibiting a detectable large scale structure across the entire target region. Note that the density gradient extends over a scale of 100 sq deg which at the median redshift of z = 0.1 equates to a 1Gpc scale structure. We do not explore the nature of this trend any further but do note some orthogonal indication of extremely large Gpcscale foreground structures arising from WMAP studies (see for example Hansen, Banday &Górski 2004 andTangen 2010).

Managing spectroscopic incompleteness
Spectroscopic incompleteness across the survey is also unlikely to be uniform but dependent on observing conditions (i.e., extinction, zenith angle etc) as well as the flux of the object in question. These cannot be decoupled and must be treated simultaneously. Here we divide the region into cells of regular sky coverage of 1 sq.deg (containing on average 75 galaxies). Within each cell we then determine the completeness as a function of apparent magnitude in 0.25 mag bins. Each galaxy, i, within this cell with a known redshift, zi, is then assigned a weight, Wi. This is the number of galaxies in some apparent magnitude bin, N (mi), divided by the number with known redshift in the same bin, N (mi, zconffinal > 0.95) i.e., Wi = N(m i ) N(m i ,zconffinal>0.95) . The global, or mean, weight for a cell (i.e., integrated over magnitude) is not particularly meaningful but can be obtained from the average assigned weight within each cell. Fig. 2 shows the mean weight in 1 sq.deg cells indicating that the incompleteness is not strongly clustered but distributed fairly randomly across the region except for a central swathe of exceptionally high completeness introduced by virtue of DR7 data. 1.8 Figure 2. The mean weight (1/completeness, see colour bar scale) within 1 sq.deg cells across the test region, indicating some minimal structure in the spatial completeness but no obvious extended regions of poor incompleteness. Note that the few hotspots of high incompleteness are aligned with the holes in the SDSS survey and represent less than 1 per cent of the data at this resolution.

Creating suitable test particles
Having established a sub-region and a mechanism of correcting for the spectroscopic incompleteness as a function of apparent magnitude and spatial location we now require a set of test particles to start exploring the cosmic variance within the test region. We adopt as test particles the most common galaxy in a locally observed sample, i.e., M * ±1.0 mag galaxies. These are both numerous and readily detectable to large distances. Selecting brighter systems potentially introduces additional variance as brighter sources are known to cluster more strongly (i.e., atypical; c.f., Norberg et al. 2002b), whereas low luminosity sources would restrict the depth of our volume and their intrinsic clustering properties are less well known. We adopt M * r − 5 log 10 (h0.7) = −21.58 mag taken from the recent ugrizY JHK LF estimates of Hill et al. (2010). The r-band is used as this is the filter in which the SDSS main spectroscopic sample is selected. Fig. 3 shows the distribution of absolute magnitude versus redshift with the systems within our defined volume shown in green. In deriving the absolute magnitudes we adopted universal k(z) and e(z) corrections (k(z) = 1.263z + 0.895z 2 − 0.566z 3 and e(z) = 2.5 log 10 [(1 + z) −0.5 ]). The choice of k(z) and e(z) corrections are not critical nor is the assumption of universal rather than individual corrections as firstly the corrections are small at low-z (z < 0.1, see Hill et al. 2010), and secondly we will always compare cells constructed over identical redshift ranges. From Fig. 3 we can see that the maximum redshift we're capable of sampling, due to the SDSS spectroscopic limit, is z = 0.1. Fig. 4 shows the variance along the line of sight by counting the space-density of test particles (i.e., M * ± 1.0 galaxies) as a function of redshift, both cumulative (mauve curve) and differential (green data points with errorbars) distributions are shown. The differ-ential counts show that the SDSS appears to suffer from an extreme underdensity locally (z < 0.02) followed by a strong overdensity at (z = 0.022) with the cumulative density well behaved from z = 0.03 to z = 0.1 (within 10% of the mean density in this range, see red lines on Fig. 4) after which incompleteness (i.e., traditional Malmquist bias) starts to affect the sample (r.f. Fig. 3). We therefore adopt a z range of 0.03 to 0.10 (minimum/maximum transverse co-moving scale of 2.2/7.3 Mpc/deg) which contains 100117 test galaxies distributed over an area of 5150 sq.deg with a co-moving radial length of 291h −1 0.7 Mpc, a lookback interval of ∼0.9Gyr, and a volume of 3.7 × 10 7 h −3 0.7 Mpc 3 . In the results that follow we will therefore only be sensitive to cosmic variance on scales below 10 7 h −3 0.7 Mpc 3 , however as cosmic variance is generally seen to be decreasing the loss of sensitivity to variance on scales greater than this volume is not expected to be a significant issue. In what follows the radial co-moving distance is typically comparable or larger (∼ 290h −1 0.7 Mpc) than the transverse lengths (< 250h −1 0.7 Mpc) and is therefore not the dominating dimension contributing to the cosmic variance. This would not be the case if the depth was quantised more discreetly, hence for what follows we have two caveats: (1) We are not sensitive to the cosmic variance on volumes larger than 10 7 h −3 0.7 Mpc 3 (2) We require that the shortest co-moving lengths defining the volume are tangential to the line-of-sight. In general this will be the case if the depth interval is ≥ 0.9Gyr in lookback time.
Finally Fig. 6 shows the histogram of the counts in 1 sq. deg cells for our test particles indicating a well behaved  distribution somewhere between an ideal Normal and log-Normal distribution -typically in previous studies a Normal distribution is assumed and we therefore follow convention. We elect to quantify cosmic variance using the simplest measure of standard deviation defined as: ζCos.Var.(%) =

RESULTS
Having defined an extensive test region (see Fig. 5) containing test particles which reflect the underlying local struc- Overlaid is a canonical Normal and logNormal distribution. Our data appears to lie somewhere between a perfect Normal and a perfect logNormal distribution indicating that our choise of statistic, the variance, is an appropriate reference measure.
tures contained in the volume, we can now sample the variance by repetitively extracting counts in fixed sized cells (truncated square-based pyramids) at random locations and repeat for increasing cell sizes. In the next two subsections we firstly explore the basic variance in square cells from 1 sq.deg to 2048 sq.deg, and secondly the variance in rectangular cells where the aspect ratio is varied from 1:1 to 1:128. In the sample extractions which follow we only sample individual test particles once, resulting in a fully uncorrelated measure of the cosmic variance but a reduction in the statistical accuracy for the larger samples and aspect ratios where fewer independent samples can be constructed. Fig. 7 shows the (cosmic) variance (data points) derived from our test region. This is shown as a percentage versus volume sampled. The range of individual values measured is shown by the grey bars and the accuracy to which the mean variance is measured is shown as errorbars. We can see that the variance decreases steadily from 60 per cent for volumes of 10 4 h −3 0.7 Mpc 3 to 10 per cent at our sampling limit of 10 7 h −3 0.7 Mpc 3 . We fit a simple second order polynomial to the data and find a good (< ±1 percent) fit given by:

The variance in square regions
where ζCos.Var represents the cosmic variance for a volume, V , as a percentage (i.e., 100∆N   N ). This expression can be used to provide a robust estimate of the cosmic variance for any given square shaped z < 0.1 survey given the total volume sampled.  Figure 7. Cosmic variance as a percentage versus volume surveyed. The data points show the mean measurements of independent volumes with the error on this mean indicated by the errorbars. The grey line denotes the range of variances about the mean. The mauve band shows our 2nd order polynomial fit to these data. The red line shows the empirical result one gets if one uses cuboids rather than pyramids for the same volume (see discussion in Section 3.4). The dark blue line shows the expected variance from Poisson statistics alone.

Contiguous versus sparse sampling
Given the scale of cosmic variance it is worth asking whether it can be more easily overcome by multiple independent sight-lines rather than a single contiguous survey. To explore this we build up a larger survey by combining N independent regions each of x sq.deg to obtain a survey of area N x. Fig. 8 shows the results as red lines originating from the building block area (x). Unsurprisingly the cosmic variance of a larger survey is significantly reduced if comprised of multiple smaller blocks rather than a single contiguous survey. For example for a survey of 32 sq.deg constructed from multiple 1 sq deg. blocks the cosmic variance is reduced from 31 per cent to 11 per cent. The cosmic variance empirically decreases with multiple sight-lines by √ N , as one would expect if the sightlines are indeed decoupled. This holds regardless of the base survey area (i.e., independent of x). Hence for multiple sight-lines Eqn. 1 can now be modified to: where N is the number of independent sight lines each of volume V . Similarly the cosmic variance can be determined for independent regions of differing area and combined assuming Poisson statistics as long as the blocks are fully independent and non-contiguous.

Dependence on aspect ratio
Survey shape and in particular the aspect ratio, or window function, is likely to impact on the cosmic variance. Not all The red lines scale exactly as one would expect from dividing by root-n where n is the number of independent observations of that specific resolution. Note that the origin of the red curves may be offset from the large data points due to uncertainty introduced by the random sampling of the parent distribution. surveys will be square/circular and may have significantly different dimensions in length and width. For example the Millennium Galaxy Catalogue (Liske et al., 2003;Driver et al 2005) has a width of 0.5 deg and a length of 75 sq deg giving an aspect ratio of 1:150. Similarly the GAMA survey (Driver et al. 2009;Baldry et al. 2010) consists of 3 chunks of 4 deg by 12 deg regions for an aspect ratio of 1:3. This aspect ratio, when taken to the extreme, can help significantly to reduce cosmic variance. Observationally long thin strips are often easier to observe because of the Earth's rotation and are more robust to cosmic variance however for the same reasons are less suitable for studies of large scale structure. Fig. 9 shows the outcome of modifying the aspect ratio by reproducing Fig. 7 for various aspect ratios ranging from 1:1 to 1:128. The cosmic variance follows an almost identical trend as for Fig. 7 but offset in amplitude. Note that the data becomes noisier for higher aspect ratios because of the limited number of independent samplings possible. For very large volumes the aspect range no longer appears to provide a gain in cosmic variance, this is because the longer tangential length approaches and exceeds the depth of the test volume resulting in a cap to the gain in cosmic variance as any volumes cosmic variance is always dominated by the two shortest lengths. In these cases the data points are shown as open symbols and the connecting lines as dotted. Finally we can incorporate the aspect ratio into Eqn. 2 by simply allowing the amplitude to vary, thus: where X is the aspect ratio, e.g., 128 for 1:128. Eqn. 3 now provides a robust estimate of the cosmic variance for the interval z < 0.1 for almost any survey in terms of the sampling volume, V , the aspect ratio of the survey window, X, and the number of independent volumes, N .

Generalising over all redshift for any survey
Extrapolating the current method beyond z ≈ 0.1 becomes non-trivial for a number of reasons: firstly, and foremost, the clustering signature of the population is evolving, with the galaxy population expected to be less clustered towards higher redshift, secondly one needs to consider the three dimensional volume shape, (i.e., in the earlier section we kept the redshift baseline constant at ∆z ∼ 0.07 which equates to a physical co-moving distance of ∼ 291h −1 0.7 Mpc). The first of these issues cannot be addressed using the SDSS and, as no suitable dataset exists at higher-z, it is currently empirically intractable. However one can adopt the values from Eqn. 3 as a robust upper limit. The issue of survey shape is also itself problematic in two ways: firstly as one moves to higher-z, for a fixed survey window, the volume becomes less conic/pyramidic and more cylindrical/cuboid due to the tendency towards a nearly constant angulardiameter-transverse length relation; secondly the freedom allowed by modifying all three dimensions of the sampling volume makes the derivation of a direct empirical expression valid beyond our maximum length impossible. In a future paper (Robotham & Driver, in prep.) we will address the first of these by providing an online Cosmic Vari-ance calculator which allows the user to specify the precise dimensions of their survey cuboid upto approximately 300×300×250h −3 0.7 Mpc 3 volumes. However this may still not cover the very deep pencil beam surveys where one might wish to bin results over radial co-moving lengths greater than 300h −1 0.7 Mpc. However we can make two relatively simple assumptions to provide an analytical workaround.
(1) Over scales greater than ∼ 250h −1 0.7 Mpc one expects no correlation of structure (although note our apparent detection of weak structure over a 1Gpc linear scale across the entire SDSS). Certainly if one axis is significantly greater than the other two one expects the cosmic variance across the longer dimension to contribute least to the total variance. Hence the cosmic variance in long cuboids should scale according to Poisson statistics if the long radial length is increased/decreased over a range > 200h −1 0.7 Mpc. i.e., a volume of 5.3 × 5.3 × 500.0h −3 0.7 Mpc 3 should have √ 2 less variance than that defined by 5.3 × 5.3 × 250.0h −3 0.7 Mpc 3 .
(2) We equate any survey volume to a cuboid where we preserve the total volume, ∆z range, and aspect ratio and derive the appropriate transverse lengths. Thus our 7320h −1 0.7 Mpc volume sampled by 1 sq.deg to 0.03 < z < 0.1 can be equated to a cuboid of dimensions 5.015 × 5.015 × 291h −3 0.7 Mpc 3 .
The first of these assumptions is reasonable assuming the radial length is always the greater of the three cuboid sides. The second can be tested by extracting equal area and equal radial length square based pyramids and cones. Adopting a constant radial length of 291h −1 0.7 Mpc we reproduce the initial results shown on Fig. 7 by now sampling our volume with cuboids (red line) rather than the original square-based pyramids (data points). To derive our cuboids we fix the total volume, the ∆z range and derive the required transverse co-moving lengths. The red curve closely follows the original data points indicating no correction is required for the change in geometric shape.
Given these two caveats one can now trivially determine an approximate cosmic variance for any survey volume. We achieve this by replacing the survey volume, V , with the product of the median redshift transverse lengths (for the survey bin in question), A & B, and a radial depth C all expressed in h −1 0.7 Mpc, then, assuming that C > 250h −1 0.7 Mpc, we find from Eqn. 3 and the caveats above that: Note that the ratio of the transverse lengths (A/B) at the median redshift has replaced the Aspect ratio (X) in Eqn. 3. For conic/cylindrical surveys one can replace A and B with (π)R where R is the transverse radius of the survey at the median redshift. This equation is strictly valid over the range 3 × 10 3 to 1 × 10 7 h −3 0.7 Mpc 3

Comparison to predictions from simulations
In a recent study Moster et al. (2010) derived cosmic variance values from a purely numerical/analytical route using simulations coupled with the adoption of a Halo Mass Function. We reproduce on Table. 1 columns from their Table. 5 for galaxies of mass 10 10.75 M⊙, the approximate turn-over point of the stellar mass function (see Baldry, Glazebrook & Driver 2008), and compare to our predictions using Eqn. 4 for the GOODs, GEMs, and COSMOS surveys. In general the values shown in Table. 3 paint a remarkably consistent picture of the cosmic variance with estimates based on the two entirely distinct methods producing remarkably consistent values for z < 2.39. One interesting offset is the tendency for the empirical method to produce lower cosmic variance values at higher redshift. While this initially appears counter-intuitive it can be explained in the context that while general clustering decreases as a function of redshift it presumably increases for fixed stellar mass. In other words the 10 10.75 M⊙ systems at high-z are destined to become the highly clustered superluminous population at low redshift. This is borne out if we compare our cosmic variance values to the lower mass values of Moster et al. at higher redshift. For example for the cosmic variance in a GOODS field in redshift interval 3.58 to 4.00 for stellar masses of 10 9.25 M⊙ Moster et al find σRMS = 0.296 consistent with our value of 30.6%. We therefore conclude that our method continues to provide a reasonable estimate of the M * cosmic variance at any redshift and that the discrepancy between Moster et al. and our work at high redshift is fully explained by the change in M * stellar mass as a function of redshift. One should therefore use our formulae if the M * point is sampled or the Moster et al. formulae for know stellar mass ranges.

COSMIC VARIANCE VALUES FOR SPECIFIC SURVEYS.
The main purpose of this paper is to derive a simple credible path to empirically based cosmic variance estimates for recent, ongoing, and upcoming surveys. The two resulting equations provide corrections for z < 0.1 robustly and for z > 0.1 under the following caveats: (1) Te derived cosmic variance is for M * ± 1 mag population only and assumed not to evolve with lookback time -this is clearly incompatible with our understanding of the evolution of structure and hence beyond z ∼ 1 the derived values should be taken as indicative only.
(2) That above 250h −1 0.7 Mpc cosmic variance scales with radial co-moving length according to Poisson statistics.
The two equations are then used to determine cosmic variance values for a number of recent, ongoing and planned surveys.