{'fraction_non_alphanumeric': 0.023603461841070025, 'fraction_numerical': 0.015950218153207924, 'mean_word_length': 4.5155818540433925, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Verifying Heaps' Law Using Google Books Ngram Data

This article is devoted to the verification of the empirical Heaps law in European languages using Google Books Ngram corpus data. The connection between word distribution frequency and expected dependence of individual word number on text size is analysed in terms of a simple probability model of text generation. It is shown that the Heaps exponent varies significantly within characteristic time intervals of 60-100 years. 1. Introduction The dependence of lexicon size on a given text size has been studied and widely discussed among linguists and other scholars in the 20th century and is still topical. The most important results obtained in this area relate to Zipf's and Heaps' laws. Heaps' law describes the dependence of individual word number in a text on a given text size and says, in its original formulation, that the number of these words increases as a square root of a number of words in a text. In this paper, we verify Heaps' law in regard to large diachronic text corpora and study the connection between Heaps' and Zipf's laws by performing statistical analyses of a large text corpus from the Google Books Ngram project.


Introduction
The dependence of lexicon size on a given text size has been studied and widely discussed among linguists and other scholars in the 20th century and is still topical. The most important results obtained in this area relate to Zipf's and Heaps' laws. Heaps' law describes the dependence of individual word number in a text on a given text size and says, in its original formulation, that the number of these words increases as a square root of a number of words in a text. In this paper, we verify Heaps' law in regard to large diachronic text corpora and study the connection between Heaps' and Zipf's laws by performing statistical analyses of a large text corpus from the Google Books Ngram project.
According to Zipf's law, word usage frequency is defined as a power function   r p r~, where r is the word rank, in other words the number of the word in a numbered list of words ranked according to their frequency [1]. Zipf's law is closely connected to Heaps' law [2], which says that the size of lexicon N (the number of particular words in a text) in a text or set of texts of size L is determined by k L N  . Different probability models of text generation (under the assumption that Zipf's law is fulfilled) result in a simple relation between the exponents β and k: Initially, Zipf's law was tested on a relatively small corpus of texts where the values of exponent β were close to 1. Heaps' law was initially derived from the analysis of news items. At that, the exponent k was estimated to be close to 0.5 [3].
Further surveys suggested different generalisations of these laws, including a general case of power dependence. It should also be noted that Heaps' law was formulated (and verified) using text corpora which created during a relatively short period of time. When Heaps' law is applied to large diachronic text corpora additional considerations are necessary.
The release of the large Google Books Ngram corpus, which contains frequency data on 8 languages over a large period of time offered an opportunity to study the static dependencies of words usage [4,5,6]. The corpus size is great. For example, it contains 2.94•10 6 English books for a period 1900-2008 which in turn include about 2.39•10 11 words. As for the Russian, German and French languages the corpora are also very large. Thus, for these languages there are respectively 3.34•10 5 , 3.49•10 5 , and 2.59•10 5 books and 2.58•10 10 , 2.50•10 10 , and 2.26•10 10 words. Zipf's law is tested in detail on the texts written in the main European languages in [6,7,8]. It is shown here that Zipf's law in its classical form is not fulfilled. Rather, a combination of two different power law distributions in different segments of the frequency distribution plot is found. The exponent of the first section (for frequently used words) is close to 1 and the exponent of the second section (for non-frequent words) is much higher and varies from 1.7 to 2.5 for different languages. The last values correlate better with the power values of Heaps' law and expression (1).
This does not mean that the problem of selecting the most suitable and simple model of word usage frequencies is solved. As shown in [6], although the model combining two different power laws fits the empirical data better than the previous models, it has to be rejected for any reasonably selected significant level.
A typical word frequency distribution is shown in Figure 1. The figure shows the word frequency distribution for English (both British and American) words in year 2000. The approximation of the empirical data using power law dependence for the rank ranges 3-440 and 1•10 4 -5•10 5 is also shown in the figure. The fitting was done using maximum likelihood estimation under the assumption that the frequency vector is governed by a multinomial distribution. It can be seen that at some segments the power law is not strictly fulfilled. The dependence is rather complicated and is not likely to be adequately described by a simple model with a small number of parameters. It should be noted that the curve goes up in the area of 500-2•10 4 ranks. Consequently, when changing the limits used to perform fitting, the values of the power exponent obtained will show a rather large range of variation. The frequencies in the range 3-300 coincide well with a power law. The power exponent, however, is significantly higher than 1 (the exponent is 1.0766 for the case shown in the figure). Using alphabetic and syllabic word-building Markov models, values which are strictly larger than 1 can be obtained [9,10,11]. The model coincides best with the empirical data in the low rank area which is rather surprising because it is normal to expect that probability models describe rare words and neologisms better than them. The frequency distribution within for large ranks is very important for the   L N dependency explanation. It should be noted that the power law for frequencies from this area cannot be regarded as 3 reliable. The available empirical data do not contradict the assumption about a sub-exponential law which can also be proved using the Markov word-building models [10,11].
In [6], a model is used to explain the empirical data which assumes an increasing probability of repeated words and a division of the lexicon into a core and a periphery. Let us consider a simpler model of text generation. We assume the existence of a (finite or infinite) set of possible words whose usage probabilities at a subsequent stage is a-priori known and equal to pi. How many words will be used on average at least once in a text of length L? To answer this question we will use the method of indicators. Consider a random variable which takes the value 1 if the i th word from our lexicon is used at least once and value 0 if it is not used at all. The probability that the i th word will not be used at least once is , consequently our random variable takes value 1 with the probability and its mathematical expectation is equal to this number. The mathematical expectation of such random variables is the average number of N words. This yields the following: Thus, we can assess the expected dependence   L N at the given word frequencies using the model described above.  Figure 2 shows the dependence of the lexicon size on the text size according to the Google Books Ngram corpus data for English (British and American combined). The number of different word forms used in the given year (ignoring capitalization) and the total number of words contained in the text base were calculated. Only word forms consisting of the letters of the Latin alphabet were considered. An approximation of the empirical data using power law dependency is also shown in the figure. The power exponent calculated using the least squares method was 0.5503. As it can be seen, the empirical data is poorly described by the exponential function. It should be noted that the correlation with the power law for the other languages presented in the Google Books Ngrams are even worse than for English.

Comparison of modelled and empirical data
The modelling of the expected lexicon size was carried out using formula (2). It is necessary to know the word-usage frequencies to calculate the expected lexicon size using formula (2). The calculation of frequencies of rare words is relatively problematic especially regarding words used in earlier time periods because the number of texts in Google Books Ngram where they can be found is rather small. Empirical frequencies of words used in the year 2000 were analysed to estimate probabilities pk because the biggest number of texts is dated back to this year. Thus, we can estimate the frequencies of the greatest number (3.97•10 6 ) of unique words. The counting results according to formula (2) are represented in the figure by the dashed line. It can be seen that the modelled dependence is close to the empirical one but it lies slightly above it. To enable a better match of the modelled and experimental data, the model was improved.

Figure 3. Change of functional words percentage in the English language
Traditionally the lexicon is divided into content and function words. The latter indicate a grammatical relationship and their frequency directly depend on the sentence structure. The share of function words in the Google Books Ngram corpus for English is shown in Figure 3 (the list of words described in [12] were used). As it can be seen in the figure, the percentage of function words can be relatively high. For example, the percentage of function words in the English language changed from about 0.48 to 0.57 in 1800-2000. It can be assumed, that all the function words will be used in a set of texts covering the wide range of topics and the number of content words will grow up with the increase of the text size. This results in the following modified model: where Nserv is the number of function words, I is the set of content word numbers in a common list, and ζ is the part of content words in a text. In the modelling process, the parameter ζ is defined for each year according to the Google Books Ngram data, after that the expected lexicon size is calculated using formula (3). The modelling results are shown by the solid line in Figure 2.
It can be seen that the model (3) yields the best approximation of the empirical data. The exponent obtained during the fitting procedure of the modelled curve using the power dependence (on the segment from 10 3 to 10 10 ) is 0.5674. This is close to the value mentioned above obtained during the fitting of the empirical data. At the same time, a serious divergence of the modelled and empirical curves can be seen. For example, variations in the large values domain can hardly be explained using a simple model.
The observed divergence could be due to imperfect model or may result from the dynamic processes in a language. The analysis of the data was performed for different time intervals to verify which of these possibilities is in play. The data selected using a 50 year sliding time window were fitted by power dependence. The dependence variations of Heaps' exponent with the time obtained for the English, Russian, German, and French languages are shown in figure 4. Two features of the obtained curves should be mentioned. First, a descending trend is typical of all the languages. Secondly, quasi-periodic variations with characteristic time periods of 60-100 years can be seen in the plots. To be sure that the power exponent variations which we can see in the figure are not erroneously obtained during the processing, we approximated the data by power dependence for the selected periods individually. Four years were selected which corresponded to the extreme points of the Heaps exponent time dependence, as well as 51-year intervals whose midpoints correspond to the selected years. The results are shown in Figure 5. Empirical values are shown by different markers which relate to different time intervals and their approximation by power dependence is shown by the lines. As can be seen from the figure, a change of the power exponent is unquestionable.
On average the number of texts in the Google Books Ngram base is much higher for later time periods. So the question is: can the observed variation of the Heaps exponent over time be due to the increase of the average size of the corpus or is it due to some dynamic processes in the language? The modelling results of the expected Heaps exponent at different text size are shown in figure 6. The modelling was performed using empirical word-usage frequencies and the modelled frequencies correlated with the Zipf law. The dependence N(L) is approximated using the power law A comparison of Figure 4 and Figure 6 shows that the Heaps exponent decreases as the corpus size increases. But both model (2) and model (3) can (in contrast what is shown in Figure 4) only give monotonically decreasing dependencies for the Heaps exponent. Thus, the quasi-periodic variations of the Heaps exponent observed in figure 4 are most likely due to dynamic processes in the language.

Conclusion
In summary we can observe that Heaps' law is fulfilled restrictedly for small texts and texts pertaining to short historical periods. The phenomenon of   L N dependence saturation should be taken into account in case of large text corpora. Lexical dynamics should also be considered in case of diachronic text corpora. The analysis of the empirical data presented in the Google Books Ngram corpora shows that the Heaps exponent varies with characteristic time interval of 60-100 years, something which must reflect dynamic processes in the language.
This work was supported by the Russian Foundation for Basic Research (grant №12-06-00404а).