{'fraction_non_alphanumeric': 0.03798128890379813, 'fraction_numerical': 0.0181519997018152, 'mean_word_length': 4.878615249780894, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Color Overmodification Emerges from Data-Driven Learning and Pragmatic Reasoning

Speakers' referential expressions often depart from communicative ideals in ways that help illuminate the nature of pragmatic language use. Patterns of overmodification, in which a speaker uses a modifier that is redundant given their communicative goal, have proven especially informative in this regard. It seems likely that these patterns are shaped by the environment a speaker is exposed to in complex ways. Unfortunately, systematically manipulating these factors during human language acquisition is impossible. In this paper, we propose to address this limitation by adopting neural networks (NN) as learning agents. By systematically varying the environments in which these agents are trained, while keeping the NN architecture constant, we show that overmodification is more likely with environmental features that are infrequent or salient. We show that these findings emerge naturally in the context of a probabilistic model of pragmatic communication.


Introduction
Overmodification describes the well-attested tendency of speakers to supply more information than apparently needed for a listener to identify the intended referent. Consider the example in Figure 1 (top). The speaker's goal is to communicate the target image (here highlighted with a yellow box) to a listener. While the utterance "circle" would suffice for the listener to pick out the target from the context, speakers often use "red circle" instead, whereas the unmodified noun "circle" is more likely when circles are typically red (middle) or when color in general is not as salient (bottom). In this paper, we show that such patterns arise naturally in neural network speaker agents that are (1) trained in these different environments and (2) reason pragmatically about a listener.
The rate of overmodification with color terms has been shown to vary in structured ways (Pechmann, 1989;Engelhardt et al., 2006;Koolen et al., 2011). Non-color-diagnostic objects, which occur in a variety of colors equally frequently (e.g., cups), are readily overmodified (Sedivy, 2003;Rubio-Fernández, 2016). For objects that are associated with one particular color (e.g., bananas), the overmodification rate drops when the color is typical ("banana" for a yellow banana), and increases when it is atypical ("blue banana" for a blue banana) (Westerbeek et al., 2015;Degen et al., 2020;Sedivy, 2003;. In addition to color diagnosticity, the visual salience of color has also been argued to play an important role in why Figure 1: Neural speakers are exposed to different environments during learning and tested on their choice of color overmodification. Varying the underlying data distribution shapes neural speakers' overinformative usage of color, explaining previous findings from human production and suggesting additional factors that may give rise to overmodification. speakers tend to frequently overmodify with color but not with other modifiers such as size (Sedivy, 2003;van Gompel et al., 2019;Davies & Katsos, 2013;Rubio-Fernandez et al., 2019;Rubio-Fernández, 2016;Brown-Schmidt & Konopka, 2011). This intuition predicts that overmodification rates decrease with reduced visual salience.
A variety of computational models have been proposed to account for overmodification patterns. The Incremental algorithm (Dale & Reiter, 1995) predicts color overmodification whenever it serves to disambiguate from at least one distractor. The probabilistic referential overspecification model (PRO;van Gompel et al., 2019) predicts that speaker preferences and visual salience explain the overmodification patterns by arguing that atypically colored objects are more salient. Recently, several models of overmodification have been proposed within the Rational Speech Acts (RSA) framework (Frank & Goodman, 2012;Goodman & Frank, 2016), in which overmodification arises from abstract speaker agents reasoning about internal listeners (Cohn-Gordon et al., 2019;Waldon & Degen, 2021). Degen et al. (2020) propose an RSA-based model that predicts overmodification patterns by defining pragmatic reasoning over non-deterministic seman-tics. In this way, the typicality effect, for instance, arises from the perception that the utterance "banana" applies less well to blue bananas than yellow ones.
What could lead a language learner to adopt the sort of semantics that produces such overmodification patterns? All previously proposed models assume that the speaker is already equipped with a complete semantics, abstracting away the process through which an agent extracts attributes of an object from raw visual input. These models therefore cannot provide a perspective on how learning to use language in the world might give rise to such a semantics in the first place. We overcome this limitation by introducing a neural network that learns the semantics from data consisting of raw visual and linguistic input. This allows us to show how overmodification arises directly from the underlying data distribution.
Our neural network-based approach also allows us to explore the role of pragmatic reasoning in overmodification. To do this, we evaluate a neural pragmatic speaker model (RSA speaker) and a neural literal speaker model (literal speaker) while varying the underlying data distributions they learn from. Both the literal speaker and the RSA speaker receive the context in the form of images as visual input, and do not observe any systematic overmodification during training. In a series of experiments, we manipulate the data that the models learn from by inducing color typicality (Exp. 2) as well as decreasing the salience of color (Exp. 3), and we compare these results to a baseline distribution (Exp. 1). We conclude that both pragmatic reasoning about a listener and properties of the underlying data distribution play a central role in achieving human-like rates of color overmodification. This provides a candidate explanation for the underlying processes giving rise to color overmodification in human language.
Even within the domain of color adjectives, overmodifica-tion rates vary based on how closely associated the object is with the presented color, i.e., the object's color-diagnosticity. Based on object recognition experiments, Tanaka and Presnell (1999) motivate a distinction of objects with high and low color-diagnosticity. Low-or non-color-diagnostic objects can plausibly occur in many colors (e.g., cups), whereas highly color-diagnostic objects only have a few highly associated colors (e.g., bananas). While speakers often overmodify with color when referring to non-color-diagnostic objects, they rarely do so with color-diagnostic ones that are presented in their typical color (Sedivy, 2003). Moreover, the rate of overmodification has been shown to increase linearly with decreasing typicality of the color for the object (Westerbeek et al., 2015;Degen et al., 2020). For instance, a yellow banana would receive the least, a blue banana the most, and a brown banana an intermediate rate of overmodification. While this literature rigorously explores color overmodification, the question of how changes in the visual environment of a learning agent causally drive overmodification remains elusive to empirical methods. In Exp. 1 and 2, we explore how the general phenomenon of overmodification arises naturally from learning from distinct data distributions. Since these data distributions can be arbitrarily adjusted, they also offer the opportunity to simulate speakers in worlds distinct from ours and study their overmodification patterns (Exp. 3).

Rational Speech Acts (RSA) framework
The Rational Speech Acts (RSA) framework is a family of computational models of communication (Frank & Goodman, 2012) where two rational agents, a speaker and a listener, are modeled as recursively reasoning about each other when producing and interpreting utterances. This work follows important precedents from game theory (Lewis, 1969;Jager, 2007;Franke, 2009;Golland et al., 2010), and has been shown to capture a wide range of linguistic phenomena (e.g., Goodman and Stuhlmüller, 2013;Kao et al., 2014;Hawkins and Goodman, 2017;van Tiel et al., 2021).
The probabilistic nature of the RSA framework lends itself naturally to integration with neural network systems. Recent work has shown that RSA reasoning helps in a variety of areas, including continual language adaptation , color reference in context (Monroe et al., 2017(Monroe et al., , 2018, following navigational instructions (Fried et al., 2018), and context-aware image-based text generation (Andreas & Klein, 2016;Vedantam et al., 2017;Cohn-Gordon et al., 2018;Nie et al., 2020).
While previous work has compared the overall performance of neural RSA speakers versus non-RSA speakers (White et al., 2020;Andreas & Klein, 2016;Monroe et al., 2017), we specifically focus on the extent to which such agents rely on overmodification (specifically with color) to communicate. We further extend previous work by investigating how changes in the underlying data distribution that the models learn from affects pragmatic language use. This leads to an account in which overmodification emerges naturally from the visual and distributional properties of the world Figure 2: Context conditions, varying in the type of information that a referring expression must provide to distinguish the target. Color overmodification occurs when color is mentioned in the shape needed condition. that a pragmatic speaker is exposed to during learning.

Task
We explore the production of referring expressions in simple reference games in which a speaker and listener are presented with the same set of three images, varying in order. The speaker's goal is to communicate the identity of the target image to the listener, whose goal is to pick out the target.
The games are generated synthetically using the Shape-World dataset creation framework (Kuhnle & Copestake, 2017). In this setup, six colors are randomly paired with four potential shapes, resulting in 24 unique color-shape compositions. Each generated image displays one colored object that is sized and positioned randomly against a black background. Each generated game consists of three distinct images and varies in the minimal amount of information the listener would need to single out the target, resulting in the four context conditions exemplified in Figure 2. Each game is paired with the most concise utterance that establishes unique reference, using a vocabulary of 11 words: those corresponding to the six colors and four shapes, plus the word shape.
The dataset's synthetic nature allows for highly controlled interventions on inherent properties of the world (e.g., color salience, Exp. 3), which can be difficult if not impossible to simulate with human learners in the real world.

Models
We experiment with two different types of speakers, an RSA speaker and a literal speaker with context. The RSA speaker reasons explicitly about the listener to generate utterances that optimize for successful communication. The literal speaker with context learns to directly map contexts to referring expressions. Based on the utterance produced by a speaker, the evaluation listener chooses one of the referents based on literal semantics learned from non-contextual data.
Evaluation listener The listener's goal is to pick out the speaker's intended target referent among three potential referents based on the speaker's utterance. To do so, the listener L Eval constructs a probability distribution over the referents that encodes L Eval 's belief about which is the intended target. More specifically, given utterance u and referents r 1 , r 2 , r 3 , L Eval will select referent r k with probability where L is a neural-based semantic function mapping utterances to referents that is learned from data.
Given an utterance u and a referent r, L outputs a probabilistic semantic value p ∈ [0, 1] that represents the listener's judgment about how well u applies to r. The probabilistic semantic value is computed as follows: given an utterance u and a referent r, where f L is an image encoder that maps an image to a ddimensional vector, g is an utterance encoder that maps an utterance to a d-dimensional vector, and σ : R → (0, 1) is the sigmoid function. RSA speaker The RSA speaker S RSA chooses a referring expression by recursively reasoning about a simulated listener's interpretation. The speaker does so by considering how each utterance would be interpreted by an internal listener model L RSA , which shares the same formulation as L Eval (Eq. 1). After considering each possible utterance, the speaker chooses the most effective and efficient utterance, i.e., the one with the highest utility. An utterance's utility U is defined as a trade-off between maximizing the likelihood of the listener identifying the correct target r t and minimizing the cost C(u) of producing utterance u: We define C(u) := λ|u| to be a linear penalty on the length of an utterance, where |u| denotes the number of words u contains. We choose λ = 0.01.
Once S RSA has computed the utility for each possible utterance, it produces the utterance that maximizes the utility: S RSA (t, r 1 , r 2 , r 3 ) = arg max u∈U U(u|t, r 1 , r 2 , r 3 ). (4) L RSA also learns semantics from data, encapsulated in its own semantic function L. The learned nature of the semantic function introduces the potential for variation. To improve the robustness of L RSA , we follow Wang et al. (2021) and model L as the mean of an ensemble of n semantic functions {L (1) , · · · , L (n) }. All models in the ensemble are identical in architecture to L Eval . We choose n = 9.
Literal speaker with context The literal speaker with context S Lit is an image-to-text model with an encoder-decoder architecture. The image encoder f S produces a vector representation of the context, i.e., the three referents. The decoder takes as input the output of f S along with the target index to generate an utterance. More specifically, for a given reference game with referents r 1 , r 2 , r 3 , and a target index t ∈ {1, 2, 3}, S Lit obtains an embedding of the reference game, h, by first encoding the three referents individually with f S and concatenating the encodings with the target index t: Then, receives h as input and generates an utterance: Here, f S is an image encoder with the same architecture as f L , and is a GRU trained using the autoregressive language modeling objective with teacher forcing, which uses crossentropy loss to penalize deviations from the ground-truth.
Training and implementation We adapted the neural module implementations of White et al. (2020). Each model was trained for 100 epochs with a batch size of 32, and optimized using Adam (Kingma & Ba, 2015) with a learning rate of 0.001 for the literal speaker and 0.01 for the semantic functions. For the literal speaker, we used validation accuracy as the model selection criterion. For the semantic functions, we used validation loss as the model selection criterion. 1

Experiment 1
We first investigate the degree to which the literal and RSA speakers choose overinformative referring expressions when all colors are equally likely to occur with all objects. This imitates the distribution of non-color-diagnostic objects, for which color overmodification is common (Sedivy, 2003).
Setup We created a dataset of 75, 000 reference games, in which all color-shape combinations appear at a uniform frequency. 55, 000 (≈ 73%) of these were used to train the neural agents, and the remaining were reserved for evaluation.
The reference games used for training were split into 11 subsets of equal size: one for training the literal speaker S Lit , one for training the semantic function L in L Eval , and nine for training the ensemble of semantic functions in S RSA 's internal listener L RSA . The games were generated from the same distribution, but no two models were trained on identical data.

Results
We start by noting that both speaker models have successfully learned to communicate the target to the evaluation listener with high accuracy (S Lit : 93%, S RSA : 94%). With this, we can turn to inspect the overmodification strategies the speaker models employ to achieve this accuracy. 1 All code and analyses for this paper can be found at github.com/feifang24/overmod-from-pragmatic-learning. Error bars correspond to 95% confidence intervals across 5 random initializations during training.
To investigate speakers' color overmodification behavior, we focus on the rate of color in addition to shape mention (e.g., "blue square") when color is neither sufficient nor necessary to identify the target (i.e., shape-needed condition in Figure 2). While strictly unnecessary, both speakers use color to overmodify, but the RSA speaker does so at a much higher rate (see Figure 3, left). The results are borne out as a significant main effect of the speaker on the rate of overmodification in a linear regression model (β = .37, SE = .01, p < .001). This suggests that overmodification with color is specifically encouraged in speakers that explicitly reason about listeners. 2 While the literal speaker remains a black box, we can inspect the RSA speaker's internal listeners to investigate what gives rise to the rate of color overmodification. More specifically, we can look at which visual features of the target are most or least likely to prompt the speaker to overmodify, and relate these patterns to the listener's understanding of these features. We define uncertainty in the listener's interpretation as the extent to which the outputs of the listener's probabilistic semantic function L deviate from the outputs from a truth-conditional semantics, which are 1 when the utterance applies to a referent and 0 otherwise. 3 Figure 3 (right) shows substantial variation in feature uncertainty, despite balanced training data-a finding we return to in Exp. 3.
In alignment with prior work, we find that overmodification rates are highest where there is high uncertainty for shape and low uncertainty for color . S RSA overmodifies with color most frequently when the target is an ellipse, which is associated with the highest uncertainty among all shapes (Figure 3 bottom right). S RSA overmodifies with color least frequently when the target is gray, which is asso- Finally, we observe that when both color and shape are necessary for unique identification of the target (both needed condition, Figure 2), S RSA and S Lit achieve comparable listener accuracy (90.43 ± 0.20% and 90.31 ± 0.56%, respectively). However, when color is strictly not necessary (shape needed) but S RSA overmodifies to a high degree, S RSA achieves a higher listener accuracy than S Lit (87.56 ± 0.34% and 85.32 ± 0.97%, respectively), borne out by a significant interaction between speaker and context condition (β = .02, SE = .01, p < .01). This suggests that overmodification might be an effective strategy that the RSA speaker leverages to improve communicative accuracy.

Experiment 2
Exp. 1 establishes that both the RSA speaker and the literal speaker generate referring expressions that overmodify with color, although at different rates. The speaker models were trained on underlying data where each color was uniformly paired with each shape, which is a commonly used feature distribution for models trained on synthetic data (e.g., Shape-World (Kuhnle & Copestake, 2017) or CLEVR (Johnson et al., 2017)). Conceptually, the uniform distribution of features makes color non-diagnostic for all object types (Tanaka & Presnell, 1999). In our second experiment, we investigate how the speaker models' overmodification changes when certain objects are more likely to occur in some colors than in others. This setup imitates the distribution of colordiagnosticity in the real world; humans refer to such objects overinformatively when they are atypically-colored but less so when typically colored (Westerbeek et al., 2015;Sedivy, 2003;Degen et al., 2020;.
Setup We repeat the experimental protocol from Exp. 1 with a new dataset, TYPICALITY, in which the frequency of certain color-shape combinations deviates from uniform. Specifically, 90% of all target circles are red, making circles in any other color atypical. We then examine how the overmodification behavior of a speaker trained on TYPICAL-ITY differs from that of a speaker trained on the dataset from Exp. 1, which will constitute the UNIFORM condition.
Results Figure 4 shows how the data distribution impacts each speaker's frequency of color overmodification when referring to a circle. Qualitatively, both speakers' rates of overmodification decrease as the color becomes more typical (red in this case) and vice versa, resembling the empirical findings from prior literature (e.g., Westerbeek et al., 2015). However, as in Exp. 1, S RSA overmodifies significantly more frequently than S Lit , regardless of object typicality. These observations are borne out in a linear regression analysis as main effects of speaker (S RSA vs. S Lit ; β = .63, SE = .02, p < .001) and color of the circle (red vs. non-red; β = −.11, SE = .02, p < .001), as well as a significant interaction (β = −.39, SE = .02, p < .001).
We now again turn to investigate what gives rise to the overmodification behavior in S RSA by examining its internal listeners' semantic functions, which represent the listeners' judgment about how well an utterance applies to a referent. When a listener is trained on TYPICALITY, the listener believes that "circle" applies well to a circle when it is of a typical color (i.e., red) and poorly otherwise ( Figure 5). In contrast, when the listener is trained on the UNIFORM data, the listener perceives circles of all colors similarly. In other words, the listener's semantic function significantly changes with the color-shape distribution it is exposed to during learning. This is borne out in a linear model as a significant interaction between dataset and red vs. non-red circles (β = −.76, Figure 6: Example reference games from the HIGH-SALIENCE (Exp. 1) and LOW-SALIENCE conditions, respectively, both with the ground-truth utterance "red circle." SE = .03, p < .001). The results suggest that the overmodification asymmetry observed as a typicality effect can be captured simply by exposing a learning pragmatic agent with a non-uniform color distribution.
The results of the listener investigation are in line with the assumed semantics in prior work where the unmodified utterance (e.g., "banana") is mapped with higher certainty to typically colored objects (yellow bananas) than to atypically colored ones (blue bananas) .
In summary, Exp. 2 shows that properties of the underlying data distribution play an important role in a speaker's patterns of overmodification. While the overall rate in S RSA is more compatible with previously observed human data, both speakers exhibit the same qualitative pattern of overmodification.

Experiment 3
The previous experiments explored whether color overmodification patterns from prior literature naturally emerge from neural speakers learning in certain environments where feature frequencies vary. The structured manipulation of the learning data led to changes in the learned uncertainties over features, which gave rise to the expected overmodification patterns. However, variation in uncertainty cannot simply be reduced to differences in frequency (e.g., see the uncertainty variation between gray and red in Figure 3). We now explore how overmodification changes when we leave the overall data frequency constant but make color more difficult to identify, rendering it less salient 4 . We thus generated a novel hypothetical world where an object's color is determined by only a single colored pixel (right context in Figure 6). Following prior work, we expect that the reduced salience should reduce overinformative color use (Davies & Katsos, 2013;Sedivy, 2003;Rubio-Fernández, 2016;van Gompel et al., 2019).
Setup We use the dataset from Exp. 1 as the HIGH-SALIENCE condition and construct a novel variant of it, the LOW-SALIENCE condition (see Figure 6). While a shape's color in the high-salience condition can be determined by randomly sampling any pixel within the shape, color in the low-salience condition needs to be extracted from one specific single pixel that is randomly placed within the shape. 4 Salience is generally underspecified. Here, we take a simplistic approach and operationalize salience as ease of feature extraction. Results As seen across experiments, S RSA overmodifies with color more frequently than S Lit regardless of the salience of color in the data distribution they learn from (Figure 7). However, in this experiment, S RSA and S Lit even come apart qualitatively. As color becomes less salient, S RSA 's frequency of color overmodification drops, while S Lit 's increases. These observations are borne out in a linear regression analysis with main effects of speaker (β = .37, SE = .01, p < .001), salience (β = .04, SE = .01, p < .001), as well as their interaction (β = −.20, SE = .01, p < .001).
The behavior of S RSA is consistent with prior work suggesting that speakers might especially overmodify with color because of its salient nature (van Gompel et al., 2019;Rubio-Fernández, 2016;Davies & Katsos, 2013;Sedivy, 2003). While S RSA 's overmodification patterns are aligned with prior work, S Lit 's are at odds with it.
Upon inspecting the S RSA 's internal listener interpretations, we see that color modifiers are associated with about 10 times more uncertainty when the listener learns from an environment with low color salience (0.094 ± 0.063) than from one with high color salience (0.008 ± 0.002). This difference is borne out as a main effect in a linear model (β = .09, SE = .01, p < .001). The results are consistent with the findings from Exp. 1, which show that the RSA speaker overmodifies infrequently when the redundant information is associated with high uncertainty .

Conclusion
When referring to objects in the world, speakers often choose to mention an object's color even when it is not necessary for uniquely picking it out among competitors (e.g., Pechmann, 1989). We have shown that a range of these attested overmodification patterns emerge naturally in neural speaker agents that learn semantics from data and reason pragmatically about internal listeners. Our findings align with color overmodification patterns reported for human speakers, and help to deepen existing modeling results concerning these patterns. More generally, we hope to have shown that neural networks are powerful tools for exploring how learning in different environments can shape pragmatic language use.