{'fraction_non_alphanumeric': 0.030664111324056328, 'fraction_numerical': 0.007666027831014082, 'mean_word_length': 4.693548387096774, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Efficient human-like semantic representations via the Information Bottleneck principle

Maintaining efficient semantic representations of the environment is a major challenge both for humans and for machines. While human languages represent useful solutions to this problem, it is not yet clear what computational principle could give rise to similar solutions in machines. In this work we propose an answer to this open question. We suggest that languages compress percepts into words by optimizing the Information Bottleneck (IB) tradeoff between the complexity and accuracy of their lexicons. We present empirical evidence that this principle may give rise to human-like semantic representations, by exploring how human languages categorize colors. We show that color naming systems across languages are near-optimal in the IB sense, and that these natural systems are similar to artificial IB color naming systems with a single tradeoff parameter controlling the cross-language variability. In addition, the IB systems evolve through a sequence of structural phase transitions, demonstrating a possible adaptation process. This work thus identifies a computational principle that characterizes human semantic systems, and that could usefully inform semantic representations in machines.


Introduction
Efficiently representing a complex environment using words is a major challenge for any cognitive system, whether biological or artificial [1,2]. Human languages reflect different solutions to this problem, as they vary in their word meanings. Nonetheless, they all exhibit useful semantic representations and obey several universal constraints [3,4]. This suggests that there might be a general principle that gives rise to efficient semantic representations, while allowing variability along some dimensions to accommodate language-specific needs. Such a principle could advance our understanding of possible forces that may shape natural languages and could potentially be used to inform useful human-like semantic representations in machines. Here we suggest that languages compress percepts into words through the Information Bottleneck (IB) principle [5].
IB is a general method for efficiently extracting relevant information that one variable contains about another. It was originally used to quantify and identify semantic relations between words [6,7], and was also suggested as a principle for learning efficient representations in biological neural networks [8,9,10] as well as in artificial neural networks [11,12]. However, so far it has not been clear how to use these applications of IB to gain a better understanding of how human-like semantic representations emerge from a need to communicate about the environment. From a cognitive perspective, a need for efficient communication is emerging as a leading principle for explaining word meanings across languages [13,14]. However, this cognitive approach has not previously been cast in terms of an independently motivated computational framework that is applicable to many machine learning tasks. Here we bring these two approaches together, and formulate the IB principle as a communication game between two agents, in which word meanings are grounded in human perception.
We present evidence that this computational principle gives rise to human-like semantic representations by studying how human languages around the world categorize colors. This is an important case study in cognitive science [4], which also has applications in machine learning [15,16]. Our primary data source is the World Color Survey (WCS), which contains color naming data from 110 languages of non-industrialized societies [17]. Native speakers of each language provided names for the 330 color chips shown in Figure 1. We also analyzed color naming data from American English [18] against the same stimulus array.  For each language l we estimated a color-naming distribution q l (w|c), where w is a color term and c is a color chip, by the empirical distribution obtained from averaging over the responses of all participants of language l (see data rows in Figure 4 for example).

Communication model
We consider a communication game between a speaker and a listener, where the messages that the speaker wishes to communicate are distributions over the environment ( Figure 2). We describe the environment by a set of objects, Y, that can be perceived by both parties, and define a meaning by a distribution m(y) over Y. Given m, the speaker may think of a particular object in the environment, Y ∼ m(y). She would like to communicate m so that the listener could think about the environment in a similar way.
We assume a cognitive source that generates intended meanings for the speaker. This source is defined by a distribution p(m) over a set of meanings, M, that the speaker can represent. The speaker communicates her intended meaning m by producing a word w taken from a lexicon of size K. We allow her to pick words according to a non-deterministic naming policy, q(w|m). This policy can be seen as an encoder because it compresses her meanings about the environment into words. The listener receives w and interprets it asm based on her decoder, q(m|w). Since this work concerns the efficiency of color naming, we assume an ideal listener that deterministically interprets w as meaninĝ m w (y) = m∈M m(y)q(m|w). Notice thatm w is the posterior distribution of Y given w. Perceptually grounded color meanings. To account for color naming data, we restrict the environment to the WCS palette. We assume that each color chip corresponds to a unique meaning, m c . Following a similar approach as [13,19], we ground these distributions in existing models of human color perception by representing colors in the 3-dimensional CIELAB space. We assume that each m c is an isotropic Gaussian in this space, namely m c (y) ∝ exp − 1 2σ 2 y − c 2 . The scale of these Gaussians reflects the level of perceptual uncertainty. We take σ to be a distance in which two colors can be distinguished comfortably, and determined it based on the results reported in [20].
Estimation of the cognitive source. In many cases it is not clear what process generates meanings for the speaker. A natural method for estimating the source distribution is by the least informative prior, which is closely related to reference priors in Bayesian inference [21,22]. For each language l we evaluated the reference prior p l (c) with respect to its naming data. These priors vary across languages, and may reflect different communicative needs [23]. However, to simplify our model and reduce the number of parameters, we assume a single cognitive source that is shared among all languages. This source is defined by averaging over languages, i.e. by p(m c ) = 1 L l p l (c) where L is the number of languages.

Information-theoretic bounds on semantic efficiency
From an information-theoretic perspective, an efficient encoder minimizes its complexity by compressing the intended message M as much as possible, while maximizing the accuracy of the interpretation M (Figure 2). In the special case where messages are distributions, this optimization problem is captured by the Information Bottleneck (IB) principle [5,24]. Notice that the communication model defined in section 2 corresponds to the Markov chain Y − M − W −M , where Y reflects how the speaker thinks about the environment. The IB principle in this case is min where I q (M ; W ) corresponds to the informational complexity of the speaker's encoder, I q (W ; Y ) corresponds to the informativeness of the communication, and β is the tradeoff between them. The informativeness term is directly related to the ability of the listener to accurately interpret M since Every language l, defined by an encoder q l (w|m), attains a certain level of complexity and a certain level of accuracy. These two quantities are plotted one against the other on the information plane shown in Figure 3. The IB curve (black) is the theoretical limit defined by the set of artificial languages that optimize Eq.(1) for different values of β. When β → ∞ each m c is mapped to a unique word, and when 0 ≤ β ≤ 1 the solution of Eq.(1) is non-informative, i.e. I(M ; W ) = 0, which can be achieved by using only a single word. In between, as β increases from 1 to ∞, the effective lexicon size of the artificial IB languages changes.

Results
If human languages are shaped by a need to maintain IB efficient representations, then for each language l there should be a tradeoff β l for which l is close to the optimal F * β l , namely ∆F β l = F β l [q l ] − F * β l is small. A natural way to predict β l is by β l = argmin β ∆F β . To evaluate the similarity between the artificial IB language defined by q β l (w|m) and natural language defined by q l (w|m), we use a generalization of the normalized information distance [25] to soft clusterings, called gNID [26]. To control for overfitting and to challenge the ability of our approach to generalize to unseen languages, we performed 5-fold cross validation over the languages that are used for estimating the cognitive source. In addition, we consider as a baseline for comparison a similar model in which all settings are the same but efficiency is evaluated according to the principle proposed by Regier, Kemp and Kay in [13]. We refer to this alternative model as RKK+. Both IB and RKK+ measure accuracy by , however in RKK+ complexity is measured by the number of frequent color terms (here we take the log). In addition, RKK+ evaluates each language w.r.t. to its optimal solution at the same complexity. We therefore consider also a variant of our IB approach, which we call C-IB, in which β l is estimated such that the IB complexity measure is constrained in the same way. That is, in C-IB it holds that I(M ; W ) is the same for q l and for q β l . We evaluate the deviation from optimality for all three models by ε l = 1 β l ∆F β l , where in RKK+ and C-IB this measure is reduced to the difference in accuracy regardless of β l . Table 1 shows the results of the 5-fold cross validation. IB and C-IB achieve very similar scores, although gNID is slightly better for IB. In addition, C-IB achieves 74% improvement in ε l and 55% improvement in gNID compared to RKK+. Similar results are obtained when the cognitive source is estimated from all folds. Therefore, the IB curve and RKK+ bounds shown in Figure 3 are evaluated for the source distribution estimated from the full data.   Figure 3 and the small ε l score for IB show that the efficiency of color naming in all languages is near the information-theoretic limit. In addition, IB's low gNID score suggests that natural color naming systems are similar to the artificial IB color naming systems. This is also supported by a visual inspection of the data (Figure 4).
We wish to emphasize that the qualitatively different solutions along the IB rows in Figure 4 are caused solely by the small changes in β. This single parameter controls the complexity, accuracy and effective lexicon size of the IB encoders. The IB categories evolve through a sequence of structural phase transitions as β increases, in which the number of distinguishable color categories changes. This process is similar to the deterministic annealing procedure for clustering [27,28]. This demonstrates a process in which the artificial IB languages may adapt to changing conditions, that also resembles cognitive theories of language evolution [e.g. 4,29].

Summary
We have shown that a need to maintain information-theoretically efficient semantic representations can account for how natural languages represent colors; the same principle could also be used to inform human-like semantic representations of color in machines. The generality of our methods suggests that this approach may also be applied to other perceptually-grounded semantic domains. The only component in our framework that is specific to color is the meaning space.