{'fraction_non_alphanumeric': 0.044210160961429996, 'fraction_numerical': 0.021692915093930322, 'mean_word_length': 4.480266286257727, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 13, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bayesian nonparametric estimation of Tsallis diversity indices under Gnedin-Pitman priors

Tsallis entropy is a generalized diversity index first derived in Patil and Taillie (1982) and then rediscovered in community ecology by Keylock (2005). Bayesian nonparametric estimation of Shannon entropy and Simpson's diversity under uniform and symmetric Dirichlet priors has been already advocated as an alternative to maximum likelihood estimation based on frequency counts, which is negatively biased in the undersampled regime. Here we present a fully general Bayesian nonparametric estimation of the whole class of Tsallis diversity indices under Gnedin-Pitman priors, a large family of random discrete distributions recently deeply investigated in posterior predictive species richness and discovery probability estimation. We provide both prior and posterior analysis. The results, illustrated through examples and an application to a real dataset, show the procedure is easily implementable, flexible and overcomes limitations of previous frequentist and Bayesian solutions.


A generalized diversity index
of the population into different species. In 1982 Patil and Taillie generalize Shannon entropy (Shannon, 1948) and Simpson's index, by far the most widely used measures of biological diversity, identifying a generalized diversity measure as the mean value for P = (P i ) i≥1 a population of relative abundances and m > 0 a parameter specifying the sensitivity to common and rare species. For m < 1 the index reduces relative differences between abundant and rare species, while for m > 1 exacerbates such differences, disproportionately favoring the most common species. Simpson's index is easily recovered for m = 2, H 2 (P ) = 1 − i P 2 i and Shannon entropy H 1 (P ) = − i P i log P i , the unique index that weighs all species exactly by their frequencies, for m → 1. Few years later, in 1988, C. Tsallis introduces (1) in statistical physics as a subadditive generalization of Shannon entropy, thus satisfying for m > 1 for P A and P B the relative abundances of two non overlapping independent classifications and m a parameter measuring the degree of deviation from additivity. Since then this index, known as Tsallis entropy, plays a significant role in non-extensive generalizations of statistical mechanics (Tsallis, 2009) and finds application in fields in which complex phenomena exhibit a power-law behaviour, reflecting a hierarchical or fractal structure. See e.g. Martins et al. 2009, Vila et al., (2011, Zhang et al. (2010) for applications in machine learning, document classification, image processing and neural signals analysis. In community ecology H m (P ) was rediscovered by Keylock in 2005 as a concave generalization of Simpson's and Shannon's measures addressing the self-similar nature of species abundances, as well as the significant amount of complex interactions between species and individuals in ecological systems. For a thorough analysis of entropy-based indices in ecology and their interpretation as diversity measures by a transformation in effective number of species see Jost (2006) and Mendes et al. (2008).
The typical problem in estimating diversity indices from a finite set of experimental data is that relative abundances are a priori unknown, and replacing them by sample relative frequencies, as in the maximum likelihood approach, produces negatively biased estimators, especially in biological communities where a large number of species has relatively small abundances and many of the rare species remain unobserved (cf. e.g. Chao and Shen, 2003). In this perspective the Bayesian approach to diversity estimation has been already advocated as a more suitable solution. Under the hypothesis of finite and known number of species a first result for Shannon entropy estimation under symmetric Dirichlet priors is in Gill and Joanes (1979). Independently in 1995 Wolpert and Wolf provide posterior first and second moments under uniform prior on the finite dimensional simplex devising a technique to obtain analogous results for more general priors. Under the same hypothesis, in the setting of information theoretical analysis of neural responses, Nemenman et al. (2002Nemenman et al. ( , 2004 show that symmetric Dirichlet priors impose a too narrow prior on Shannon entropy and suggest to use as an alternative a specific mixture of those priors. As for Tsallis diversity a recent and exhaustive analysis of maximum likelihood estimation compared to computationally intensive estimation methods is in Butturi-Gomes et al. (2014) while, to the best of our knowledge, the unique Bayesian proposal is under uniform prior on the finite dimensional simplex (Holtse et al., 1998).
Here we present a fully general Bayesian nonparametric solution, under a large class of priors, to the problem of estimating the general index H m (P ) when the relative abundances of the species in the population are unknown, the number of species is unknown and possibly countably infinite and the size of the sample available from the population is small and so is the number of different species observed. Preliminary contributions along those lines, providing explicit posterior mean and variance under the two-parameter extension of the Poisson-Dirichlet distribution (Pitman and Yor, 1997), are in Cerquetti (2012) and Archer et al. (2013), respectively for Simpson's and Shannon's index. Here not only we are able to generalize those results to the whole family of Tsallis indices, but we derive posterior moments under a large class of priors introduced in Gnedin and Pitman (2006), which extends the family of twoparameter Poisson-Dirichlet distributions while conserving the Gibbs product form of the corresponding exchangeable partition probability function. This class, which is commonly referred to as the Gibbs priors class, has become extremely popular in modern Bayesian nonparametrics as a tractable generalization of the Dirichlet process prior (Ferguson, 1973). Related Bayesian estimation in species sampling applications has been devised in Lijoi et al. (2007Lijoi et al. ( , 2008 and largely addressed in Favaro et al. (2009Favaro et al. ( , 2012bFavaro et al. ( , 2013. See also Cerquetti ( , 2013Cerquetti ( , 2013b. Nevertheless, until now, the focus has been on posterior predictive species richness and discovery probability estimation. Here we provide the first results in this general setting for a large family of measures of diversity. Notice that despite (1) is defined for any m > 0, we will restrict to the case m ∈ N . We start by briefly introducing the Gnedin-Pitman class and some of its main properties.

Gnedin-Pitman priors
Given an infinite random discrete distribution P = (P i ) i≥1 , then the law of the infinite exchangeable random partition Π n = {A 1 , . . . , A k } of [n], for n ≥ 1, induced by sampling from P is given by where n i = |A i |, (i 1 , . . . , i k ) ranges over all ordered k-tuples of distinct positive integers and (P i ) i≥1 is any rearrangements of the ranked atoms (P ↓ i ) i≥1 of P . See Pitman (2003Pitman ( , 2006 for exhaustive accounts on exchangeable random partitions. Gnedin-Pitman priors are the largest class of infinite random discrete distributions with corresponding exchangeable partition probability function (EPPF) (2) in the Gibbs product form for α ∈ (−∞, 1) and V = (V n,k ) weights satisfying the backward recursive relation where V 1,1 = 1 and (x) y = (x)(x + 1) · · · (x + y − 1) is the usual notation for rising factorials. Given (n 1 , . . . , n k ) the multiplicities of the first k species observed in a random sample of size n = i n i , the probabilities to observe the jth old species or a new yet unobserved species at step n+1 are easily obtained from (3) and correspond respectively to p α,V (n + j ) = V n+1,k (n j − α)/V n,k and p α,V (n k+1 ) = V n+1,k+1 /V n,k . By Theorem 12 in Gnedin and Pitman (2006) each random partition belonging to the class (3) is a probability mixture of extreme partitions, namely: finite symmetric Dirichlet partitions for α < 0, Ewens (θ) partitions (Ewens, 1972) for α = 0, and Poisson-Kingman conditional partitions driven by the stable subordinator (Pitman, 2003) for α ∈ (0, 1). Therefore each element of the Gnedin-Pitman priors family can be obtained by mixing corresponding extreme random discrete distributions. Mathematical tractability characterizes some specific subfamilies of (3) that are typically the most widely implemented in the modern Bayesian nonparametric literature. Consequently first we provide closed form general expressions for prior and posterior moments of Tsallis measures under general Gnedin-Pitman priors and, as a by product, of Shannon's and Simpson's diversities. Then we derive explicit corresponding formulas under two-parameter (α, θ) Poisson-Dirichlet priors, and its particular cases, normalized (α) Stable and (θ) Dirichlet priors, and under the two-parameter Gnedin-Fisher priors (Gnedin, 2010, Cerquetti, 2011b. As for the class of exponentially tilted Poisson-Kingman priors driven by the stable subordinator, and in particular for normalized Inverse Gaussian and normalized generalized Gamma priors, which belong to the Gnedin-Pitman class for α ∈ (0, 1), (cf. Pitman, 2003), the explicit derivation of prior and posterior moments of Tsallis diversity and Shannon entropy from our general results will be the topic of a future paper.
We stress that the technique adopted here allows to obtain the full sequence of prior and posterior moments of H m (P ) under (α, V ) Gnedin-Pitman priors. Nevertheless we give explicitly just the first three moments for H m (P ) and the first two moments for H 1 (P ).

Prior analysis 2.1 Tsallis diversity
One of the main problem in Bayesian estimation of Shannon entropy under symmetric Dirichlet distributions on the relative abundances, is that this class induces an extremely concentrated distribution on the prior belief, with variance that vanishes as the number of species becomes large (cf. Nemenman et al. 2002Nemenman et al. , 2004. The very same problem arises under two parameter (α, θ) Poisson-Dirichlet priors on the infinite dimensional simplex for θ large (cf. Archer et al. 2013). To check for analogous pathological behaviours induced by priors belonging to the Gnedin-Pitman class on the distribution of H m (P ) for m ≥ 2, we first derive explicit closed form expressions for the first three prior moments.

Shannon entropy
As for Shannon entropy H 1 (P ), the next theorem generalizes to the entire Gnedin-Pitman family the results on prior first and second moments under symmetric Dirichlet priors and under two-parameter Poisson-Dirichlet priors, already obtained respectively in Gill and Joanes (1979) and Archer et al. (2013).
Theorem 2. Given a random discrete distribution P = (P i ) i≥1 , distributed according to a (α, V ) Gnedin-Pitman model with EPPF (3), then ∂m 2 V rm,s and ψ 1 (·) the trigamma function then The derivation of the third moment follows along the same lines. For brevity we omit here the explicit derivation.
Remark 1. If the distributions of the size-biased atoms (P j ) j≥1 of the specific Gnedin-Pitman prior are known explicitly, like e.g. when a stick-breaking construction of the (1 − V i ) has been devised, then first and second prior moments of Shannon entropy can also be obtained through the same route adopted in Archer et al. (2013) under two-parameter Poisson-Dirichlet priors. This is also the case, for example, of normalized Inverse Gaussian priors, for which the stick breaking construction has been recently obtained in Favaro et al. (2012). Notice in fact that in the proof of Theorem 2 and, by properties of size-biased distributions (see e.g. Pitman, 1996Pitman, , 2003, forP 1 and P 2 the first and second size-biased atoms, Nevertheless the implementation of this technique could be difficult when raw and mixed moments of log(P 1 ) and log(P 2 ) are not known in closed form. The results in Theorem 1 and Theorem 2, while being valid for the entire Gnedin-Pitman class, allow to derive prior moments for Shannon entropy directly by the Gibbs weights.

Examples
Example 1 (Two parameter Poisson-Dirichlet (α, θ) priors). For α ∈ (0, 1) and θ > −α the two parameter (α, θ) Poisson-Dirichlet model (Pitman, 1995, Pitman andYor, 1997) has EPPF in the Gibbs form for (x) y↑α = x(x + α) · · · (x + (y − 1)α) generalized rising factorials. EPPFs of Dirichlet (θ) priors and normalized (α) Stable priors arise respectively for α = 0 and θ = 0 in (10). An application of (5) and (6) yields and (12) Simpson's index prior mean and variance follow by (11) and (12) for m = 2 (see Cerquetti, 2012). Shannon entropy prior mean and variance, derived in Archer et al. (2013) exploiting the stick-breaking construction of the size-biased atoms, , follow by an easy application of (8) and (9): Prior moments of Tsallis entropy, Simpson index and Shannon entropy under Dirichlet (θ) priors and normalized (α) Stable priors follow from the previous formulas respectively for α = 0 and θ = 0.  (11) and (12). For comparison purposes, for m = 3, the index the has been standardized by normalization with the maximum value (m − 1) −1 = 1/2. We refer the reader to Archer et al. (2013) for analogous prior analysis for Shannon entropy. Notice that for m = 2 under normalized Stable prior (θ = 0) the parameter α corresponds to the prior guess on the Simpson' index of diversity. Increasing α for a given θ or increasing θ for a given α produces the same effect on the prior guess, which approaches the maximum value, and the same effect on the uncertainty, increasing the concentration of the prior around the mean. This behaviour suggests, as for Shannon index in Archer et al. (2013), that the choice of a two parameter Poisson-Dirichlet prior for the generalized Tsallis diversity H m should be confined  Example 2 (Two parameter Gnedin-Fisher (ψ, γ) priors). In 2010 Gnedin introduced another tractable two-parameter family of laws for random discrete distributions belonging to the Gibbs class for α < 0. The model is obtained by mixing the uniform prior on the finite dimensional simplex with shifted generalized Waring distributions (Xekalaki, 1983) on the number of species. Here we adopt the parametrization devised in Cerquetti (2011) for its specific tractability in the Bayesian nonparametric setting. The general form of the EPPF, for ψ ∈ [0, 1) and 0 < γ < ψ + 1, is given by and reduces to the one-parameter case for ψ = 0. An application of (5) and (6) yields Specializing (13) and (14) for m = 2 corresponding formulas for Simpson's index easily follow. As for Shannon entropy prior mean an application of (8) yields which reduces to E γ (H 1 ) = γ −1 (1 − γ) for the one-parameter case (ψ = 0). Prior second moment arises by calculating the limit for m → 1 of the first and second partial derivatives of with respect to m and then applying (9). For the sake of brevity we omit here the explicit formulas. Figure 2 shows some simulated prior distributions of Shannon entropy under Gnedin-Fisher one parameter prior for different values of γ. A peak at zero, which increases with γ, characterizes those priors since the Waring mixing distribution puts positive mass γ at ξ = 1, the case of a population with a unique species. Nevertheless the priors on H 1 appear to be sufficiently flat and uninformative in (0, 10) for small values of γ, thus suggesting that those models may provide an interesting choice as priors for Shannon entropy, not suffering of the problem of concentration of the two parameter Poisson-Dirichlet model and its particular cases.

Tsallis diversity
In the next theorem, which generalizes the results in Holste et al. (1998, Sect. 4), we obtain the first three posterior moments of H m (P ) thus providing what needed for Bayesian point estimation under quadratic loss function, for approximate interval estimation by Chebyschev's inequality and for studying the symmetry of the posterior distribution. Given the vector (n 1 , . . . , n k ) of the multiplicities of the first k different species observed in order of appearance in a sample of size n, similarly to Theorem 1, the results arise deriving the posterior moments of S m = ∞ j=1 P m j under Gnedin-Pitman priors and then applying the binomial theorem. The details of the proof are in the Appendix.
Theorem 3. Let n = (n 1 , . . . , n k ) be the multiplicities of the first k species observed in a sample of size n, then, under a general (α, V ) Gnedin-Pitman prior on the unknown relative abundances (P i ) i≥1 and Posterior moments of Simpson's index (m = 2), generalizing the results in Cerquetti (2012) to the entire Gnedin-Pitman class, arise specializing (15), (16) and (17) for m = 2.

Shannon entropy
Similarly to Theorem 2, posterior estimation of Shannon entropy under general Gnedin-Pitman priors, which provides a substantial generalization of the results in Wolpert and Wolf (1995), Nemenmann et al. (2002, 2004 and Archer et al. (2013), is obtained by repeated application of Hôpital's rule to the results in Theorem 3. The following Proposition provides posterior mean and posterior second moment under a general (α, V ) Gnedin-Pitman model. Notice that, despite it is possible to derive the closed form, the posterior second moment expression is extremely complex and not easy to be calculated in real data applications. Therefore, in Section 4, we illustrate the posterior uncertainty of H 1 (P ) deriving the highest posterior density intervals (HPD) by the simulated posterior distributions.
Proposition 1. Let n = (n 1 , . . . , n k ) be the multiplicities of the first k species observed in a sample of size n, then, under a general (α, V ) Gnedin-Pitman prior on the unknown relative abundances, and

Examples
Example 3 (Two-parameter Poisson-Dirichlet continued). An application of (15) and (16) under two-parameter Poisson-Dirichlet priors yields and For m = 2 (20) and (21) reduce to Simpson's index posterior mean and variance. (See Cerquetti, 2012 for details). Shannon entropy posterior estimation arises by an application of (18). For V n,k = (θ + α) k−1↑α /(θ + 1) n−1 then E α,θ (H 1 |n) = ψ 0 (θ + n + 1) Posterior variance may be obtained through the second posterior moment, by applying and bearing in mind that It is easy to check that the results agree with Archer et al. (2013). Like for prior moments, posterior estimation under Dirichlet priors and normalized Stable priors arises specializing the previous formulas respectively for α = 0 and θ = 0.

A real data application
The high level of generality of the results proposed in the previous sections does not allow a complete direct evaluation of our technique with respect to single alternative frequentist and Bayesian procedures already available in the literature. Nevertheless a sample illustration of the implementation of our method and a comparison with the results provided by existing procedures can be conducted for a single index in the Tsallis class and a specific prior choice. Here we apply our Bayesian nonparametric estimation of Shannon index to a dataset on tropical foliage insects from sweep samples taken in 25 sites in Costa Rica and the Caribbean Islands (Janzen, 1973) already considered in Chao and Shen (2003). The dataset consists of the frequency counts for beetles collected in day time from the site referred to as "Osa primary-hill, dry season, 1967". Frequency counts, or sampling formulas, provide an alternative codification of a realization of a random partition in terms of the vector of the number of blocks of the same size. For m j the number of species represented j times in the beetles sample, the observed dataset is given by m 1 = 59, m 2 = 9, m 3 = 3, m 4 = 2, m 5 = 2, m 6 = 2, m 11 = 1, with a total of k = 78 different species seen and n = 127 total observations. Most of the species observed have only one, two or three individuals represented in the sample, with few abundant species. Table 4.1 is an elaboration of Table 6 in Chao and Shen (2003). Those authors propose a nonparametric method combining the Horvitz-Thompson estimator adjusted for unequal probability sampling scheme and the concept of sample coverage to adjust for the presence of unseen species. They show by simulations that their estimator is preferable to previous frequentist estimators and performs well when a large fraction of the species is missing in the sample. We add the four estimates obtained applying our method under Gnedin-Fisher priors (Janzen, 1973) obtained applying formula (22), while highest posterior density intervals are derived by the posterior sampled values, by means of a specific R function. With respect to the Horvitz-Thompson estimator corrected for the number of unseen species in Chao and Shen (2003) our technique yields higher point estimates with narrower intervals, thus providing a better account of the effect of missing species with an increased precision. The information in this particular dataset greatly overcomes the information contained in the chosen prior leading to robust conclusions on the diversity of the population, independently of the choice of the prior γ. We stress here that, unlike the method proposed by Chao and Shen, the nonparametric Bayesian approach we are proposing naturally takes into account the presence of unseen species. In fact by construction the prior is placed on a theoretically infinite space of relative abundances for α ∈ [0, 1) and on a random number of finitely many species for α < 0. Then the multiplicities of the first k species in the observed sample update both the relative abundances of the seen and of the unseen species. Figure 3 shows the whole posterior distributions under Gnedin-Fisher priors of Shannon index as obtained by simulations for the beetles dataset. The bell-like shape with a very low level of asymmetry suggests that first and second posterior moments, therefore posterior mean and highest posterior density intervals, are enough to summarize posterior inference on Shannon diversity index in this case.
of Theorem 3. Let (X i ) i≥1 be an exchangeable random sequence of species observations driven by a general (α, V ) Gnedin-Pitman prior on the unknown relative abundances and n = (n 1 , . . . , n k ) the multiplicities of the first k species observed in a sample of size n, then for ξ ≥ 1 posterior moments of S m = ∞ j=1 P m j can be decomposed as follows for n = (n 1 , . . . , n k ), s = (s 1 , . . . , s k * ) and m = (m 1 , . . . , m k ). For k * = 0 then p m (n) = V n+v,k V n,k k j=1 (n j − α) m j and for v = i s i then For v = mξ and m j = mξ j from (33) and multinomial formula, where the sum is over the space of non-negative integers (ξ 1 , . . . , ξ k ) with sum ξ. For v = mξ and s i = mξ j then, from an application of (27) to the posterior partition probability function (34), and analogously from (32) where the second sum is over the space of positive integers (z 1 , . . . , z k * ) with sum l.