{'fraction_non_alphanumeric': 0.05136735269241613, 'fraction_numerical': 0.01973498731322244, 'mean_word_length': 4.090556831228473, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Shannon's entropy and its Generalizations towards Statistics, Reliability and Information Science during 1948-2018

Starting from the pioneering works of Shannon and Weiner in 1948, a plethora of works have been reported on entropy in different directions. Entropy-related review work in the direction of statistics, reliability and information science, to the best of our knowledge, has not been reported so far. Here we have tried to collect all possible works in this direction during the period 1948-2018 so that people interested in entropy, specially the new researchers, get benefited.


Introduction
The notion of entropy (lack of predictability of some events), originally developed by Clausius in 1850 in the context of thermodynamics, was given a statistical basis by Ludwig Boltzmann, Willard Gibbs and James Clerk Maxwell. Analogous to the thermodynamic entropy is the information entropy which was used to mathematically quantify the statistical nature of lost information in phone-line signals by Claude Shannon (1948). Although a similar kind of result was independently developed by Wiener (1948), the approach of Shannon was different from that of Wiener in the nature of the transmitted signal and in the type of decision made at the receiver (cf. Nanda (2006)). For more on the history of the development of entropy, in the context of thermodynamics and information theory, one may refer to Mendoza (1988).
Apart from thermodynamics and communication theory, the recent past has shown the applications of entropy in different fields, viz, economics, finance, statistics, accounting, language, psychology, ecology, pattern recognition, computer sciences, physical sciences, biological sciences, social sciences, fuzzy sets etc., making the literature on entropy voluminous. Shannon, along with several others, have shown that the information measure can be uniquely obtained by some natural postulates. Shannon's measure is found to be restrictive as discussed later. Another measure of information as proposed by Rényi (1961) is somewhat a generalized version of that of Shannon's.
As the number of papers in the field of entropy has increased enormously over the last seven decades, we feel that the time is ripe to have a review paper on the topic. Since it is nearly impossible to survey all the literature associated with entropy across different fields of theory and applications, we decide to focus on the role of Shannon's entropy and its generalizations towards statistics, reliability and information science. With this scope in mind, we identified 106 relevant articles in terms of theory and practice that were published in the last seven decades of which 44 were published post-2000 era, which clearly indicates the recent progress in this research area as well as the amount of interest the researchers are still showing in this field. The paper is organized as follows.
Section 2 gives a simple derivation of Shannon's entropy and discusses some of its important properties, followed by other related entropies. Here we also discuss joint and conditional entropies along with expected mutual information. Since Shannon's entropy is useful for new items only, its modified version is discussed in Section 3, where this can be used for any item which has survived for some units of time. Section 4 deals with cumulative residual entropy corresponding to Shannon's and some other. Entropy estimation and some tests based on entropy are discussed in Section 5. Here the Kullback-Leibler divergence is also discussed. Applications of the entropies are discussed in Section 6 whereas Section 7 gives some concluding remarks.

Notations and Preliminaries
Information may be transmitted from one person to another through different ways, viz., by reading a book or newspaper, watching television, accessing digital media, attending lecture etc. We need to have information when an event occurs in more than one way, otherwise there is no uncertainty about the occurrence of the event and hence no information is called for. As an example, we may be interested to know whether there will be rain tomorrow or not. In case we know (by sixth sense!) that there will be rain tomorrow, then the event of raining tomorrow (say, event A) is certain, and hence we do not need any further information on this. In other words, if we are not certain of raining tomorrow, there is some uncertainty about its occurrence. Once the event A or A c takes place, we are sure of having rain or not, and there is no uncertainty prevailing about its occurrence. This leads to the conclusion that information received by the occurrence of an event is same as the amount of uncertainty prevailing before occurrence of the event.

Derivation of and Discussion on Shannon's Entropy
Let us explain the concept of entropy with an example. Suppose E is the event of getting a job by a candidate. If P (E) = 0.99, say, i.e., the likelihood of getting the job is very high for the candidate, which eventually reduces the amount of unpredictability for getting the job. On the other hand, if P (E) = 0.01, the chance of getting the job is very low, resulting in high level of unpredictability. Therefore, one can conclude that the more is the chance of getting a job, the less is the entropy.
It is clear from the above discussion that if p is the probability of occurrence of an event, then the entropy of the event, denoted by h(p), is decreasing in p. Further, any small amount of additional information on the occurrence of the event will reduce the amount of uncertainty prevailing before getting the additional information. This shows that h(p) must be continuous in p. It is also obvious that h(1) = 0.
Further, if any two events E 1 and E 2 are independent with P (E i ) = p i , i = 1, 2, the information received by the occurrence of two events E 1 and E 2 together is same as the sum of the information received when they occur separately, i.e., h(p 1 p 2 ) = h(p 1 ) + h(p 2 ).
Let us transform the variable as p = a −x with some a > 0. We write Thus, we have the following axioms.
Let m be a positive integer. Then, by Axiom (iii) above, we have Writing m = n(m/n) and using Axiom (iii) again, we have This, on using (2.1), gives Thus, we have φ(x) = x.φ(1) for any positive rational number x. Since any irrational number can be written as a limit of sequence of rational numbers, the continuity of φ gives that φ(x) = x.φ(1) for any positive irrational number x. Combining the two, we say that for any positive real number x, where c = φ(1). Thus, we get Without any loss of generality, we take c = 1 and a = 2, which gives h(p) = − log 2 p.
As far as the event E is concerned, the information to be received is either h(p) or h(1 − p), and we don't know which one, until the occurrence of E or E c . Hence expected information received concerning the event E, known as entropy corresponding to E, is Generalizing this to n events with probability vector p = {p 1 , p 2 , . . . , p n } we get which is in agreement with the intuition that the maximum uncertainty prevails when the alternatives are equally likely. 2 Let X ∼ p = {p 1 , p 2 , . . . , p n }. The entropy corresponding to the random variable X, or equivalently, corresponding to the probability vector p, is denoted by H(p) (and also by H(X)). It is to be noted here that p is not an argument of H. It is a label to differentiate H(p) from H(q), say, the entropy of another random variable Y ∼ q = {q 1 , q 2 , . . . , q m }.
Below we give the postulates proposed by Shannon.
(a) H(p) is symmetric in its arguments. Although Shannon's entropy has been extensively used by different researchers in different contexts, it has some drawbacks as pointed out by several researchers including Awad (1987). He has observed that defining entropy as weighted average of the entropies of its components is not the correct way. To be more specific, if we consider the probability distribution p = {p 1 , p 2 , p 3 } = {0.25, 0.25, 0.5}, then contribution of p 1 is same as that of p 3 because 0.25 log 2 (0.25) = 0.5 log 2 (0.5), although p 1 = p 3 . He has also observed that the distributions are not identifiable in terms of entropy. To see this, let us consider the probability distributions p and q as Clearly H(p) = H(q) although p = q. It is also to be noted that, for discrete random variable, Shannon's entropy is always nonnegative whereas, its corresponding counterpart for continuous random variable, given in (2.3), may not be so. To see this, let X ∼ U (a, b). Then Another very important drawback of Shannon's entropy, as pointed out by Awad (1987), is that, for the transformation Y = aX + b, we have Clearly, (b) violates the basic idea that measuring some characteristic in two different units should not change the obtained information. To overcome the limitations of Shannon's entropy, Awad (1987) has suggested a different entropy, known as Sup-entropy, given in (2.2).

Other Related Entropies
Let p and q be two probability distributions. Then p * q is the direct product of the distributions, that is, the distribution given by p * q = {p i q j , i = 1, 2, . . . , n, j = 1, 2, . . . , m}. Rényi (1961) replaced Postulate (d) above by

The postulates (a)-(c) and (d ) result in
which is known as Rényi entropy. If p = {p 1 , p 2 , . . . , p n } is a generalized probability distribution (i.e., p i 0, for i = 1, 2, . . . , n, and n i=1 p i 1), then Rényi entropy is given by However, in our discussion we will consider only ordinary probability distributions (i.e., n i=1 p i = 1).

Remark 2.2
The following points are interesting to be noted.
• If α is very close to 0, then where n is the cardinality of the probability vector p. Hartley (1928) has shown that H(n) = log 2 (n), known as Hartley entropy, is the only function mapping from N → R satisfying It can be noted that (i) H A α and H B α are obtained from Rényi entropy by re-parametrization. To be more specific, H A α is obtained by replacing α by α − n + 1 whereas H B α is obtained by replacing α by α/n.
(ii) As motivation of re-parametrization, Varma has mentioned that, in Rényi entropy α, can be a proper fraction whereas in his entropy it is not. However, the difficulty, if any, in α being a proper fraction has not been discussed in his paper.
Then Harva and Charvát (1967) derived an entropy, known as structural α-entropy, as which satisfies the following postulates.
• S(p1, . . . , pi−1, q1, q2, pi+1, . . . , pn; α) = S(p1, . . . , pi−1, pi+1, . . . , pn; α) + p α i S q 1 p i , q 2 p i ; α , for every q 1 + q 2 = p i > 0, i = 1, 2, . . . , n, α > 0. Awad (1987) proposed an entropy, called Sup-entropy, as Next we give the definition of entropy for continuous random variable in the line of the same for discrete random variable as defined earlier. Shannon's and Rényi's entropies for continuous random variable X are defined as and respectively. Since replacing log 2 by ln (natural logarithm) is only a constant multiple of H(X), we sometimes use ln in place of log 2 . Wyner and Ziv (1969) have given an upper bound to entropy as That the converse is not true can be seen by taking the distribution of X as Cauchy. Khinchin (1957) considered entropy as for any convex function g with g(1) = 0. Clearly, g(x) = − log x gives Shannon's entropy. Pardo et al. (1995) discussed a general entropy, called (h, φ)-entropy, defined as where φ : [0, ∞) → R is concave and h : R → R is increasing and concave (or, φ is convex and h is decreasing and concave). Following entropies are obtained as a special case of (h, φ)-entropy for different choices of φ and h.
Azzam and Awad (1996) modified the Sup-entropy as where X = {X 1 , X 2 , . . . , X n } is a random sample, L is the corresponding likelihood function and θ is the unique MLE of θ. To get an idea about the relative performance of three entropies, H(θ)(= H(p)), A n (θ), and B n (θ), they have calculated the relative losses in the three entropies by approximating gamma by normal, binomial by Poisson and Poisson by normal, and observed that the relative loss is decreasing in both n and θ. They have also observed that the entropy, B n (θ), has some advantage over the entropies A n (θ) and H(θ). Now, Shannon's entropy can be alternately expressed, by writing F (x) = p, as log dp dx dp = 1 0 log dx dp dp dF −1 (p) dp dp. (2) . . . x (n) as the ordered observations of (x 1 , x 2 , . . . , x n ), an estimator of H(X) is obtained as Here we take x (i) = x (1) , for i < 1 and x (i) = x (n) , for i > n. We must mention here that if X has pdf/pmf f , then H(X) is sometimes equivalently written as H(f ).

Some Further Discussions
Let {x 1 , x 2 , . . . , x m } be the realizations of the random inputs X and let {y 1 , y 2 , . . . , y n } be those of the random outputs Y in an information channel. Suppose that an information x i will be received as output y j has probability p j|i = P (Y = y j |X = x i ), i = 1, 2, . . . , m, j = 1, 2, . . . , n. Then the matrix known as the corresponding channel matrix, is a stochastic matrix. Let P (X = x i ) = p i0 be the probability that x i is selected for transmission, P (Y = y j ) = p 0j be the probability that y j is received as output and let P (X = x i , Y = y j ) = p ij be the probability that x i is transmitted and y j is received. Then the joint entropy is the entropy of the joint distribution of the messages sent and received, and is given by The marginal entropies are given by The following lemma will be used in sequel.  On using the above lemma one can prove that with equality if and only if X and Y are independent. The conditional entropy of Y given that X = x i is defined as The average conditional entropy of Y given X is the weighted average given by It can be shown that H(X) + H(Y |X) = H(X, Y ), which means that if X and Y are observed, but only observations on X are revealed, then the remaining uncertainty about . This also says that the revelation of the observations of X cannot increase the uncertainty of Y because with equality if and only if X and Y are independent. When message x i is sent and the message y j is received, then, for i = 1, 2, . . . , m and j = 1, 2, . . . , n, the expected mutual information is defined as It can also be shown that H(X) − H(X|Y ) may be considered as the reduction in uncertainty about X when Y is revealed. So, I(X, Y ) may be considered as the amount of information conveyed by Y about X. Thus, we have that the amount of information conveyed by X about Y is same as that conveyed by Y about X. It can be noted that which is zero if and only if X and Y are independent. For more discussion on this, one may refer to Cover and Thomas (2006).

Entropy of Used Items
So far we have discussed entropy of a new item. A natural question could be -how to define entropy of a used item? In survival analysis and life testing experiments, one has information about the current age of the component under consideration. In such cases, the age must be taken into account when measuring uncertainty. Obviously, the Shannon's entropy is unsuitable in such situations and must be modified to take the age into account. Ebrahimi and Pellerey (1995) took a more realistic approach and proposed to use known as Residual Entropy, where λ F (·) is the failure rate function corresponding to the distribution F . After the component has survived up to time t, H(X; t) basically measures the expected uncertainty contained in the conditional density of (X − t) given that X > t about the predictability of the remaining lifetime. They have defined a stochastic order as follows.
for all t. 2 It is quite possible that X LU Y but X LR Y or X LR Y . The residual entropy has also been used to measure the ageing and to characterize, classify and order lifetime distributions by different researchers. Below we give corresponding definition of H(X, ·) for discrete random variable.
Then the discrete residual entropy, denoted by H d (X; k), is defined as Ebrahimi (1996) proved that, for a nonnegative continuous random variable X, H(X; t) uniquely determines the distribution of X. A similar result for discrete random variable was proved by Rajesh and Nair (1998). It was observed in Belzunce et al. (2004) that both the above results were erroneous. The correct result is given below.
Theorem 3.1 If X has an absolutely continuous (resp. a discrete) distribution and an increasing residual entropy H(X; t) (resp. H d (X; k)), then the underlying distribution is uniquely determined. 2 The following counterexample proves that the condition 'H d (X; k) is increasing' in the above theorem cannot be dropped. On using different forms of H(X; t), different distributions (viz. uniform, exponential, geometric, beta, Pareto, Weibull, logistic) were characterized by Nair and Rajesh (1998), Sankaran and Gupta (1999), Belzunce et al. (2004) and Nanda and Paul (2006a). In the context of nonparametric class based on entropy of a used item, Ebrahimi (1996) defined the following.
A random variable X (or equivalently, its distribution function F ) is said to belong to the DMRL (decreasing in mean residual life) class (resp. IMRL ( increasing in mean residual life) class) if E(X t ) is decreasing (resp. increasing) in t. Asadi and Ebrahimi (2000) proved that if X k:n is DURL, then (i) X k+1:n is DURL; (ii) X k:n−1 is DURL; (iii) X k+1:n+1 is DURL, where X k:n is the k th order statistic from a sample of size n, i.e., X k:n is the k th largest random variable in the arrangement X 1:n X 2:n . . . X k:n . . . X n:n of the random variables X 1 , X 2 , . . . , X n . They have characterized generalized Pareto distribution having survival function,F , given bȳ in terms of different expressions of residual entropy. The Rényi entropy for a used item is defined as Asadi et al. (2005) have shown that if the density is strictly decreasing (resp. increasing and finite support), then H α (X; t) uniquely determines the distribution for α > 1 (resp. 0 < α < 1). They have characterized generalized Pareto distribution in terms of H α (X; t).
Analogous to entropy of a used item (called residual entropy), Di Crescenzo and Longobardi (2002) proposed an entropy based on the random variable (X|X x) as Some characterization results based onH(t) were discussed in Nanda and Paul (2006b). A discrimination measure between (X|X t) and (Y |Y t) (analogous to Kullback-Leibler (KL) divergence measure between X and Y ) was proposed by Di Crescenzo and Longobardi They proved that if Y lr X 1 rh X 2 then, for t > 0,Ī(X 1 , Y ; t) Ī (X 2 , Y ; t). As a measure of divergence between two used items, Ebrahimi and Kirmani (1996a) proposed dynamic KL divergence given as

Other Related Results
In the Shannon's entropy, if the density f (x) is replaced by P (|X| > x) we get which is called cumulative residual entropy (CRE) by Rao et al. (2004). For a nonnegative random variable, this reduces to and its dynamic 1 version, known as dynamic cumulative residual entropy (DCRE), was studied by Asadi and Zohrevand (2007), whereF t , given byF t (x) =F (t + x)/F (t), is the survival function of the residual random variable X t = (X − t|X t). The Weibull family was characterized in terms of CRE of X 1:n , the first order statistic, by Baratpour (2010). For some more results on CRE one may refer to Navarro et al. (2010). CRE and DCRE were further modified by different researchers viz. bivariate extension of residual and past entropies by Rajesh et al. (2009), cumulative residual Varma's entropy and its dynamic version by Kumar and Taneja (2011), cumulative past entropy (replacinḡ F by F in CRE) and its dynamic version by Minimol (2017). Cumulative residual Rényi entropy (CRRE) and its dynamic version (DCRRE) was discussed by Sunoj and Linu (2012). If X is an absolutely continuous random variable with a pdf f (·), then Rényi's entropy of order β is defined as Abraham and Sankaran (2005) extended Rényi's entropy of order β for the residual lifetime X t as Sunoj and Linu (2012) have replaced f (·) in both the above expressions by the survival func-tionF (·) to define CRRE (Cumulative Residual Rényi's Entropy) and DCRRE (Dynamic CRRE) as respectively. Psarrakos and Navarro (2013) defined generalized cumulative residual entropy (GCRE) and its dynamic version as

Inference Based on Entropy
Here we shall discuss different methods of estimation of entropy and different testing problems based on entropy.

Estimation of Entropy
First, we discuss kernel density estimator of entropy, which is the most commonly used nonparametric density estimator found in the literature (see, for example, Rosenblatt (1956), Parzen (1962), Prakasa Rao (1983) among others). As defined by Rosenblatt (1956), the kernel estimator based on a random sample X 1 , X 2 , ..., X n from a population with density function f is given by where a n is the bandwidth and K is the kernel function. In practice, {a n } is chosen in such a way that a n (> 0) → 0 as n → ∞ and the kernel function K is a symmetric probability density function on the entire real line. Ahmad and Lin (1997) used this f to define entropy estimator, H(f ) = − f (x) ln f (x)dx, and proved the following consistency result.
If, along with (i)-(iv), we have Let m i be the frequency of the event E i in a sample of size N , i = 1, 2, . . . , n. Then the probability of the event E i is estimated by p i = m i N and the entropy is estimated as Basharin (1959) showed that H is biased, consistent and asymptotically normal with Basharin also proved the asymptotic normality when p i and n are fixed. If p i and n are allowed to vary then, according to Zubkov (1959), Hutchenson and Shelton (1974) gave an expression for mean and variance of the above entropy estimator based on multinomial distribution. They have shown that, for multinomial distribution, where q i = 1 − p i , and where [x] denotes the highest integer contained in x. A generalized version of H(f ), given by On using kernel estimator, Joe (1989) estimated the Shannon's entropy corresponding to a multivariate density as where f , a kernel estimator of f , is given by with h as the bandwidth, under the following assumptions.
• f is continuously twice differentiable with respect to each argument.
• k is a p-variate density function.
• Tail probabilities of f can be neglected.
The last condition was dropped in Hall and Morton (1993). It is to be mentioned here that the kernel estimator of entropy used in Hall and Morton (1993) is different from that of Joe. Hall and Morton (1993) have estimated the entropy is known as leave-one-out estimator. Now, let us define ρ(x, y) as the p-dimensional Euclidean distance between x and y. Also, for a fixed X i , define . . .
Here ρ i,k is the distance of X i and its k th nearest neighbour. Goria et al. (2005) estimated where k ∈ {1, 2, . . . , N − 1}, ψ(z) = Γ (z) Γ(z) and c(p) = 2π p/2 pΓ(p/2) . If the density function f is bounded and for some > 0, then H k,N is an asymptotically unbiased estimator of H(f ) for δ = 1, and it is a weak consistent estimator of H(f ) for δ = 2, as N → ∞. It is to be mentioned here that the residual entropy for continuous random variable has been estimated by Belzunce et al. (2001) by using kernel estimation method. Shannon (1949) found that normal distribution has the maximum entropy among all absolutely continuous distributions having finite second moment. This property, along with H mn (as defined in Equation (2.4)) was used by Vasicek (1976) to test for normality which was further shown to be less sensitive to outliers than Shapiro-Wilk (1965) W-test by Prescott (1976). Entropy was used by Dudewicz and van der Meulen (1981) for testing U (0, 1) distribution. Testing related to power series distribution, which includes binomial, Poisson, Geometric etc. as special cases, was discussed in Eideh and Ahmed (1989). Later, the idea of Vasicek (1976) was used to test for multivariate normal by Zhu et al. (1995). As defined before, Goria et al. (2005) used H k,N to construct goodness-of-fit test for normal, Laplace, exponential, gamma and beta distributions. This H k,N was also used to test for independence in bivariate case. A simulation study indicates that the test involving the proposed entropy estimate has higher power than other well-known competitors under heavy-tailed alternatives. Vexler and Gurevich (2010) used

Testing Based on Entropy
Shannon's entropy-based test statistic in the empirical likelihood ratio form, for testing f = f 0 , where F n is the empirical distribution function. They have shown that the proposed tests are asymptotically consistent and have a density-based likelihood ratio structure. This method of one-sample test was further extended by Gurevich and Vexler (2011) to develop two-sample entropy-based empirical likelihood approximations to optimal parametric likelihood ratios to test f 1 = f 2 . The proposed distribution-free two-sample test was shown to have high and stable power, detecting a non-constant shift alternatives in the two-sample problem. Let E r:k be the entropy of the r th order statistic. Then and Taking some linear combination of E 1:n and E n:n and using the concept of Vasicek (1976), Park (1999) considered the test statistic where J is continuous and bounded, with J(u) = −J(1 − u), to test for normality. Next, we define cross-entropy and its relation with Kullback-Leibler Divergence measure for discrete random variable. Suppose p is the true distribution and we mistakenly think the distribution is q. Then the entropy will be This is known as Cross Entropy, and we denote it by H p (q) (to distinguish it from H(p, q)). Note that is known as Kullback-Leibler Divergence. It is easy to see that D KL (p||q) 0. To see how the (Expected) Mutual Information is related to the KL divergence, note that where p X,Y is the joint mass function of (X, Y ), and p X and p Y are the marginal mass functions of X and Y , respectively. Also, it can be noted that Note that D KL (p||q) = D KL (q||p). So, KL divergence is not a proper distance measure between two distributions. For continuous distribution, KL divergence is defined as where f 1 and f 2 are the marginal densities of X and Y respectively. Csiszár (1972) considers a generalized version of KL divergence as for any convex g with g(1) = 0. Clearly, g(x) = − log x in the above expression gives KL divergence. Note that Equality holds iff f 1 (x) = f 2 (x) for all x. Since KL divergence is not symmetric, different symmetric divergence measures have been studied in the literature. One such measure is Burbea and Rao (1982aRao ( , 1982b proposed symmetric divergence measures based on φentropy, defined as where φ is defined on some interval I. They defined J -divergence, K-divergence and Ldivergence between x and y as respectively. Next, we define record and show its relation with KL Divergence measure. Let {X i , i ≥ 1} be a sequence of iid continuous random variables each distributed according to cdf F (·) and pdf f (·) with X i:n being the i th order statistic. An observation X j is called an upper record value if its value exceeds that of all previous observations. Thus, X j is an upper record if X j > X i , for every i < j. An analogous definition can be given for lower record values. For some interesting results on records one may refer to Kundu et al. (2009) and Kundu and Nanda (2010). Also define the range sequence by V n = X n:n − X 1:n . Let R n denote the ordinary n th upper record value in the sequence of {V n , n ≥ 1}. Then R n are called the record range of the original sequence {X n , n ≥ 1}. A new record range occurs whenever a new upper or lower record is observed in the X n sequence. Suppose that R l n and R s n are the largest and the smallest observations, respectively, at the time of occurrence of the n th record of either kind (upper or lower), or equivalently of the n th record range. Ahmadi and Fashandi (2008) showed that the mutual information between R l n and R s n is distributionfree. They have also shown that KL divergence of R l n and R s n is also distribution-free and is a function of the number of records (n) only and decreases with n.
Several applications of KL divergence in the field of testing of hypotheses, in particular, for multinomial and Poisson distributions, are discussed by Kullback (1968). The concept of KL divergence is used by Arizono and Ohta (1989) for testing the null hypothesis of normality (H 0 ) with mean µ and variance σ 2 . Taking f 2 (x) in (5.2) as the pdf of normal distribution, (5.2) can be expressed as The test statistic for testing H 0 is obtained as where I mn , an estimate of D KL (f 1 ||f 2 ), is found to be Under H 0 , it is shown that KL mn P → √ 2π, as n → ∞, m → ∞, m/n → 0. The authors have showed that the critical region for testing H 0 is KL mn ≤ KL mn (α), where KL mn (α) is the critical point for the significance level. Similarly, KL divergence measure is used for testing exponentiality by Ebrahimi et al. (1992) and Choi et al. (2004). Test for location-scale and shape families using Kullback-Leibler divergence is discussed in Noughabi and Arghami (2013). KL divergence is used for testing of hypotheses based on Type II censored data by Lim and Park (2007) and Park and Lim (2015). For some more uses of KL divergence in testing of hypotheses one may refer to Choi et al.

Applications
It has been observed that different researchers have shown usefulness of entropy in different fields. Clausius (1867) has used entropy in the field of Physical Sciences, Shannon (1948) has used it in Communication Theory, whereas Shannon (1951) has shown its usefulness in Languages. An application of entropy in Biological Sciences has been reported by Khan (1985). Gray (1990) has used it in Information Theory. Chen (1990) uses entropy in Pattern Recognition. Brockett (1991) and Brockett et al. (1995) have found its applications in actuarial science and in marketing research respectively. While Rényi entropy is used by Mayoral (1998) as an index of diversity in simple-stage cluster sampling, generalized entropy is used by Pardo et al. (1993) Bailey (2009) has used entropy in Social Sciences. Its application in Economics has been shown by Avery (2012). The entropy and the divergence measures have been used by Ullah (1996) in the context of econometric estimation and testing of hypotheses, where both parametric and nonparametric models are discussed. The application of entropy in Finance may be obtained in the work of Zhou et al. (2013). Farhadinia (2016) has shown the application of entropy in linguistics. It is observed that in analyzing imbalanced data, the usual entropies exhibit poor performance towards the rare class. In order to get rid of this difficulty, a modification has been proposed by Guermazi et al. (2018). Shannon's entropy has been used in the multiattribute decision making by Chen et al. (2018). Kurths et al. (1995) have shown different uses of Rényi entropy in physics, information theory and engineering to describe different nonlinear dynamical or chaotic systems. Considering the Rényi entropy as a function of α, H α is called spectrum of Rényi information (cf. Song (2001)). It is used by Lutwak et al. (2004) to give a sharp lower bound to the expected value of the moments of the inner product of the random vectors. To be specific, write N α (X) = e Hα(X) and N α (Y ) = e Hα(Y ) . If X and Y are independent random vectors in R n having finite p th (p > 1) moment, then for α > n n+p , where C is a constant whose expression is explicitly given in Lutwak et al. (2004). The Rényi entropy is also used as a measure of economic diversity (for α = 2) by Hart (1975), and in the context of pattern recognition by Vajda (1968). The log-likelihood and the Rényi entropy are connected as V ar(log f (X)).
Writing S f = V ar(log f (X)) we have Being location and scale independent, S f can serve as a measure of the shape of a distribution (cf. Bickel and Lehmann, 1975). According to Song (2001), S f can be used as a measure of kurtosis and may be used as a measure of tail heaviness. In order to use β 2 = µ 4 /µ 2 2 , fourth moment must exist. However, S f can be used even when fourth moment does not exist. In order to compare the tail heaviness of t 6 , the t distribution with 6 d.f., and Laplace distribution, we see that β 2 (t 6 ) = 6 = β 2 (Laplace) which tells that t-distribution with 6 d.f. and Laplace distribution are similar in terms of tail heaviness. However, S(t 6 ) ≈ 0.79106 whereas S(Laplace) = 1 which tells that Laplace distribution has heavy tail compared to t 6 distribution, which is also evident from Figure 1. Since the Cauchy distribution does not have any moment, comparison of tail of Cauchy distribution with that of any other distribution in terms of β 2 is not possible. In this case the above measure may be of use. Measure of tail heaviness for probability distributions based on Rényi entropy of used items has been studied in Nanda and Maiti (2007).

Concluding Remarks
Since the work of Shannon (1948), people have found applications of entropy in different disciplines including Linguistics, Management and different branches of Science and Engineering. The literature on entropy has been developing since last seven decades. It is almost impossible to write a review on the vast literature, especially when it branches out to different directions. In the present work, we have tried to give a brief review of entropy having applications in Statistics, Reliability and Information Science. This collection of entropy-related work will surely benefit the researchers, specially the newcomers in this field, to further the work which will enrich the related theory and help the practitioners. The entropy is developed by Shannon starting from a set of postulates. Some kind of natural modifications in the set of postulates have led to different kind of entropies which are well-fitted in some specific practical situations. In spite of its well applicability, Shannon's entropy possesses some drawbacks which have been suitably modified by different researchers. Once the Shannon's entropy has been modified to overcome its limitations, a natural question that arises is -what are the possible postulates that will lead to the revised entropy? One important and interesting problem in this direction is to find out a set of postulates that will generate different variations of Shannons entropy (suggested only to take care of the limitations). Once the postulates are obtained, one must see whether all the postulates so obtained are feasible from practical point of view. If yes, the modified entropies may remain, otherwise some essential modifications in the modified entropies have to be allowed.
We have noted that Shannon's entropy has been used in statistics for goodness-of-fit test, test of different hypotheses, estimation of distribution etc. One may take up the job of using the modified entropies for the same purpose. Since the modified entropies are improvement over Shannon's entropy in some sense, it is expected that the tests developed (or distribution estimated) based on the modified entropies will be better in some sense, which may be in terms of power of the test or anything alike.
While discussing different entropies in the direction of statistics, reliability and information sciences, some similar literature may have been dropped unintentionally and the authors are apologetic for the same.