{'fraction_non_alphanumeric': 0.039789823505636146, 'fraction_numerical': 0.01675124623882876, 'mean_word_length': 4.634615384615385, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 6, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Do ever larger octopi still amplify reporting biases? Evidence from judgments of typical colour

Language models (LMs) trained on raw texts have no direct access to the physical world. Gordon and Van Durme (2013) point out that LMs can thus suffer from reporting bias: texts rarely report on common facts, instead focusing on the unusual aspects of a situation. If LMs are only trained on text corpora and naively memorise local co-occurrence statistics, they thus naturally would learn a biased view of the physical world. While prior studies have repeatedly verified that LMs of smaller scales (e.g., RoBERTa, GPT-2) amplify reporting bias, it remains unknown whether such trends continue when models are scaled up. We investigate reporting bias from the perspective of colour in larger language models (LLMs) such as PaLM and GPT-3. Specifically, we query LLMs for the typical colour of objects, which is one simple type of perceptually grounded physical common sense. Surprisingly, we find that LLMs significantly outperform smaller LMs in determining an object's typical colour and more closely track human judgments, instead of overfitting to surface patterns stored in texts. This suggests that very large models of language alone are able to overcome certain types of reporting bias that are characterized by local co-occurrences.


Introduction
Large language models (LLMs) have been compared to hypothetical giant octopi living underwater that are exposed to a lot of language data (Bender and Koller, 2020). Such octopi would struggle to understand what actually happens on land as they lack the physical perceptual experience of living there. As such, they may overfit to text-only corpora and thus amplify reporting bias (Gordon and Van Durme, 2013) rather than faithfully reflecting the physical world. 1 https://github.com/googleresearch/language/tree/master/language/octopus-llm (code). Figure 1: On typical colour judgments, large language models (LLMs) greatly outperform small LMs which previously were found to be no better than corpus statistics (Google Ngram). See Table 2 for full results.
In textual corpora, humans do not tend to mention what is commonly known, instead using language to express new information, which is likely less common. For example, when describing the colour of a banana: "green banana" has much higher frequency than "yellow banana" in the Google Books corpus. 2 It is natural to expect LMs would overfit to such reporting bias since they are trained to memorise such co-occurrence statistics. To observe this, we can query widely used pretrained models, such as ROBERTA Large (Liu et al., 2019) with our previous example. Given the prompt "It is commonly known that most bananas have the color <mask>", ROBERTA ranks "green" the highest. 3 This agrees with corpus statistics derived from raw text corpus such as the Google Ngram (Lin et al., 2012) mentioned above. 4 Paik et al. (2021) test pretrained LMs' perception of colours and confirm that they perform no better than naive co-occurrence statistics extracted from the corpus. In fact, naively using corpus statistics achieves around 40% accuracy on their proposed colour probing benchmark CoDa while the best LM performs similarly. Zhang et al. (2022) extend the evaluation to a broader range of visual properties, confirming that reporting bias can negatively influence model performance and increasing model size does not help. Shwartz and Choi (2020) repeat the reporting bias experiments of Gordon and Van Durme (2013) on pretrained LMs and find that LMs overestimate rare events and actions, also amplifying reporting bias.
However, the LMs tested by Paik et al. (2021);Zhang et al. (2022); Shwartz and Choi (2020), i.e., GPT-2 (Radford et al., 2019), BERT (Devlin et al., 2019), ROBERTA, and ALBERT (Lan et al., 2020), usually have only several hundred million parameters and are of much smaller sizes than LLMs available now. In this work we probe T5 (Raffel et al., 2020), GPT-3 (Brown et al., 2020), andPALM (Chowdhery et al., 2022) of various sizes, with parameter counts ranging from 770M to 540B. Surprisingly, we find that LLMs almost double the performance of small language models (SLMs) on the typical colour task (Figure 1). 5 Paik et al. (2021) point out that SLMs achieve poor performance on objects that typically only have one colour (such as bananas), possibly due to their true colour being an aspect of common sense and thus not frequently mentioned in the training corpus. We find in contrast that LLMs achieve surprisingly good performance in this category, reaching >80% accuracy. After plotting accuracy against model size, we observe that scaling up is universally helpful for improving LLMs' performance on the colour probing benchmark (CoDa;Paik et al. 2021). Moreover, as LLMs are scaled their correlation to corpus ngram statistics plateau, suggesting that memorising (local) co-occurrence statistics cannot explain their success. 6 Our study presents controlled analyses on the colour prediction task as a case study to show that scaling up LLMs could overcome surface-level pat-5 For convenience and consistency, we refer to all models with fewer than 10B parameters as small language models (SLMs) while those with more than 10B parameters as LLMs. 6 A careful reader would note here that the models' training data may differ distributionally from Google Ngram. We discuss this more in §5. tern memorisation (i.e., text reporting bias in our case) and learn physical world common sense at least to some extent. This is an important and surprising finding as it provides a key evidence to counterargue the previous consensus that despite achieving better performance for a range of NLP tasks, larger LMs are more prone to overfitting to corpus statistics and therefore amplifying the reporting bias. Our study points out that this criticism on model scale is misleading as it is not based on the complete picture, and when the model capacity is increased to a significantly large scale such as PALM-540B and GPT-3 davinci , they start to overcome reporting bias and are able to abstract physical common sense from text.

Method
To test whether LLMs replicate corpus biases rather than human judgment in the typical colour task, we compare the models' output distributions with the corpora's distribution and the distribution of human judgments. Visual perception provides an ideal testbed as corpus statistics can vary from physical facts; obvious facts are left unspoken. In this case, we focus on the typical colour task, largely following the setup by Paik et al. (2021). Given a query asking the colour of an object, the model must output a distribution over eleven possible colours. We then compare the output distribution to both corpus statistics and average human judgement to examine their respective correlations.
In the following, we explain how we query the LLMs and use their predictions. We test LLMs in three setups: zero-shot, one-shot, and five-shot.
Zero-shot. We use the following prompt across all models: It is known that most {OBJECTq} have the color <mask> where {OBJECT q } is replaced with the object's name (in plural form). 7 After inputting the prompt, we compute next-token-prediction likelihood for all 11 colours in the CoDA label space and record the log-likelihood scores for all answers as the output distribution of the query: where Θ is LM's parameters; c is the color; "prompt" is the input prompt specified earlier. For 0-and 5-shot prompting, the answer scoring scheme remains the same. See Appx. §C for details of how few-shot prompts are constructed.

Experimental Setup
Dataset. The CoDa dataset contains queries and human judgments of 521 objects. For each object, CoDa has a human-perceived colour distribution over 11 basic colours in English. The 11 colours were identified by Berlin and Kay (1969) and include black, blue, brown, grey, green, orange, pink, purple, red, white, yellow. As an example of the dataset, the object "Carrot" has the humanperceived scores of black: 0.0, brown: 0.023, orange: 0.797, etc., where the scores over 11 colours sum up to 1. CoDa contains three types of questions (1) Single (2) Multi and (3) Any. "Single" means the object has only one typical colour such as "Carrot" which is typically orange. "Multi" objects have between two and four typical colours: "Apple" is frequently red or green. "Any" objects have no fixed set of typical colours, such as "Shirt" and "Car". By default we report micro-average results across all three types. However, we also discuss the "Single" category in detail as it is thought to be especially indicative of reporting bias because such facts are rarely stated in texts. The statistics of CoDa are listed in Table 1  Metrics. We use Acc @1 , ρ human , ρ ngram . Acc @1 measures whether the model gets the most typical colour of an object correct. Other metrics are useful, but less clearly interpretable: ρ human measures a set of predictions' Spearman's ρ correlation with the distribution of human colour judgments (however, there is low human consensus for some objects and colours). Higher Acc @1 is better; higher ρ human indicates a closer match to human judgments. ρ ngram measures the models' predictions' correlation with the Google Ngram statistics. Fitting corpus statistics is not necessarily good or bad: we report it to see its relationship with both model size and model performance.
Google Ngram baseline. Together with queries and human judgments, Paik et al. (2021) also provide ngram stats collected from Google Books and Wikipedia to compute the correlation with these corpora. Specifically, they consider all bi-and trigrams containing a colour followed by an object. A corpus-based baseline is then computing the accuracy/correlation between the total ngram counts of colour-object pairs and the human perceived-scores. We use Google  Compared LLMs and their sizes. OpenAI does not disclose the exact size of their text models Ada, Babbage, Curie and Davinci. According to blog.eleuther.ai/gpt3-model-sizes, they roughly correspond to 350M, 1.3B, 6.7B, and 175B, which we use as the models' parameter counts. For other models (i.e., T5 and PALM), their number of parameters are made clear in the original papers. We list all compared models' sizes in the second column of Table 2.

Results
Main results (Table 2). We show our main results in Table 2. As a general trend, LLMs with >10B parameters all significantly outperform SLMs with <10B parameters, and performance 0-shot  Table 2: Results on CoDa (average over all three types). For Acc @1 and ρ human (the higher the better), the best performing models within each model class are boldfaced. The symbol * denotes numbers from Paik et al. (2021), which uses a more optimistic protocol, aggregating the best per-object performance over 10 hand-crafted prompts.
increases monotonically with scale within each model class. While the SLMs do not perform significantly better than Google Ngram (accuracy 36.3%), LLMs achieve up to 64.9% (PALM-540B 5-shot). PALM-540B 5-shot also correlates best with human judgments. For PALM and GPT-3, few-shots are much better than 0-shot; 10 while for T5, 0-shot seems to be the best.
Results on the "Single" colour split (Table 3).
The "Single" split deserves extra attention as it has the highest human consensus and is also considered to be common sense knowledge, implying it is rarely stated in the corpus (Paik et al., 2021). While none of the SLM baselines outperform the Ngram baseline on Acc @1 , the largest PALM and GPT-3 surpass the Ngram baseline by nearly 40%. Furthermore, the LLMs' predictions correlate significantly more to human judgments. We also present an error analysis on the "Single" split in Appx. §B.5. Out of the ten errors made by PALM-540B, only one is a clear mistake where the model classifies picnic baskets as red. For other nine errors, the error seems to be associated with the ambiguous nature of the questions or the dataset construction process.
Correlation metrics (Figure 2). For GPT-3, its correlation with corpus ngram statistics (ρ ngram )  initially increases but then plateaus and even decreases (on 5-shot: 42.2 → 51.9 → 57.1 → 55.9). On PALM, ρ ngram decreases from the start as model size grows (on 5-shot: 62.0 → 61.0 → 60.1). On both models, ρ ngram initially is larger than ρ human . However, for model sizes above 10 11 parameters, both models' predictions have ρ human > ρ ngram . This suggests that when LMs are small, they can underfit corpus ngrams. When LMs start to be scaled up, they increasingly fit the corpus. However, after a certain model size, additional scale does not lead to more overfitting to corpus statistics. On the contrary, as LLMs' predictions correlate more with human judgment, they also start to decorrelate with corpus statistics.   (2022) use the counts of colour occurrences with objects in bi-and tri-grams. However, to fully understand whether LLMs overfit, we also need to consider longer contexts as it is possible that the typical colour of an object is described in longer pieces of text; thus, LLMs performance improvements can be attributed to memorising long-term dependencies better than SLMs. In this case, the "generalisation" is only memorising a context that is similar to the prompt. Alternatively, LLMs may learn good representations of the quantifiers, such as "most", and the usage of the atypical colours in the text may not co-occur with quantifiers suggesting it is common. In future work, we intend to examine whether a similar phenomenon persists when collecting occurrence stats over typical model input lengths and using more fine-grained data that also characterises premodifiers such as quantifiers.
Comparing within model class for better control of confounders. Though LLMs today are almost all Transformer-based models with similar autoregressive pretraining objectives, we note that there are caveats preventing us from having a perfect control over design choices on pretraining corpora and specific architectures. In terms of pretraining data, within-family models of different sizes generally use the same training data (GPT-3models are however less transparent in this regard). However, it is unclear what differences there are across model families. In terms of model architectures, T5 is an encoder-decoder model while GPT-3 and PALM are decoder-only models. PALM has further modifications on top of the original Transformer architecture such as using SwiGLUE activation (Shazeer, 2020) instead of the standard ReLU; using RoPE embeddings (Su et al., 2021) instead of the original relative position embeddings. As a result, more conclusive findings should be drawn within model classes, e.g. comparing PALM-540B with its two smaller versions instead of GPT-3 models.
Colours live on a spectrum. The evidence we obtain does not reflect whether LLMs have a finegrained and holistic understanding of the nature of colour. That is, colours live on a continuous spectrum. LLMs could have solved CoDa by identifying the mappings between objects and colours but not colour's relative positions on the spectrum. One way to probe this is to examine if LLMs can resolve colour synonyms (e.g., do LLMs know that "scarlet" occupies a subspan of the colour red?). However, a rigorous and systematic study of this problem is beyond the scope of this study.

Conclusion and Future Work
In this work, we examine LLMs ability to make typical colour judgments, a simple property of visual common sense. Contradicting Paik et al. (2021); Zhang et al. (2022), we find that typical colour judgments do not follow an inverse scaling law, and scale is indeed quite critical for high accuracy on the task. While generalising from this task to visual reasoning as a whole is premature, we provide some evidence that larger models of language alone are able to overcome a basic type of reporting bias. Future work will look at a wider range of physical properties (Collier et al., 2022) and more carefully control for the data and model size. We also hope our work opens an avenue for empirically verifying on what level meaning acquisition is possible from a cognitive linguistic perspective (Piantasodi and Hill, 2022). A A More Comprehensive Table  (Table 5) In the main text, we compare different models under different setups in Table 2. To provide more information for reference and also strengthen our findings, we present a more comprehensive Table 5 which also reports Kendall's τ as a correlation metric, and include Wikipedia stats provided by Paik et al. (2021) as another source of ngrams. The main conclusion remains the same. Kendall's τ has identical trend to Spearman's ρ, and similar fitting trend of Google Ngram is also shown on Wikipedia.

B Further Discussions
Here we present some more extensive discussions on several topics that concern the experimental setup, including testing SLMs under the same setup as the LLMs (Appx. §B.1); testing different prompts (Appx. §B.2); the discrepancies among analysed corpora and the real pretraining corpora of LLMs (Appx. §B.3); the risk of direct data leakage (Appx. §B.4); and error analysis (Appx. §B.5); .

B.1 Real zero/few-shot setup for SLMs
In the main text, we used SLM numbers reported by Paik et al. (2021) under an optimistic setup: i.e. out of 10 prompts, choosing always the prompt that maximises per-object's performance when evaluating models. We note that when under the same evaluation protocol as LLMs, SLMs' performance would have dropped to chance level. We pick the best performing SLM ROBERTA Large as an example. When consistently using one prompt, ROBERTA Large has only an accuracy score of 7.3%. Prompting with few-shot examples does help a bit. However, the 5-shot accuracy of ROBERTA Large (real) still has a roughly 50% gap compared with few-shot performance of the best LLMs.

B.2 LLMs' Sensitiveness to Prompts
For the main experiment, we choose an arbitrary prompt: "It is known that most {OBJECT} have the color <mask>.". However, it is possible that LLMs are particularly good or bad at this prompt and it is worth testing whether LLMs are robust to how we ask the question. In Table 6, we test GPT-3's sensitivity towards different prompts. First, we change the quantifier "most" to "all", no quantifier, "some", "few", and "no". We find that the LLM is sensitive to the quantifier and produces scores generally well correspond to the quantity being asked. Note that "all" and no quantifier lead to lower performance than "most", possibly due to the question is unnatural since there is rarely any object exclusively having only one colour. We also paraphrase the original prompt and find that a grammatical paraphrased query can lead to up to around +/-6% performance difference. An ungrammatical prompt will damage the model's performance, even including key words such as "most", "color", and "common sense".

B.3 Discrepancy among Corpora
As discussed in Limitations ( §5), we use Google Books and Wikipedia in line with Paik et al. (2021) for direct comparison. As can be seen in Table 5, Google Ngram is better agreeing with human judgment. Moreover, Google Books is much larger than Wikipedia. So, in the main experiments, we use it as an approximation of pretraining corpora. However, it remains unknown how well these sources' ngram distributions align with the real training corpora of LLMs. In future work, there should ideally be more strict control and better access to the pretraining data to draw firmer conclusions.

B.4 Have the LLMs seen test data during training?
It is unlikely that LLMs have seen the test data in its exact form in their pretraining corpora. As the whole web can be used as training data, this is a real risk. However, we think it is unlikely that LLMs have seen CoDa. The CoDa dataset was released on October 2021. GPT-3-davinci-002 was trained with data until June 2021; GPT-3-curie/babbage/ada-001 were using data until October 2019; 11 T5's pretraining corpus C4 was crawled on April 2019. PaLM's precise training data is unknown, but the paper was published after CoDa. However, performancewise PaLM is not significantly better than GPT-3davinci-002, which uses training data before the release of CoDa.

B.5 Error Analysis
Here we pick the errors made by the models on Single-type questions to understand why or what type of questions they make mistake. Both GPT-3 and PALM achieve above 80% in this category. We randomly sample 10 errors made by PALM-540b (5-shot) and list them below.  Table 4: Evaluating the best performing SLM on CoDa, using one consistent prompt (the same setup for all LLMs tested). Performance of the optimistic 10-prompt setup by Paik et al. (2021) and also performance of LLMs are listed for reference. When evaluated under the same protocol as LLMs, the best performing SLM ROBERTA's performance drops very significantly and is at chancel level. Most of the ten queries seem to be ambiguous. Black and brown scorpions are both common; the color of a mango might be described as orange or yellow; kitchen sinks are normally gray but bathroom sinks are normally white; old computer monitors are normally gray but newer ones are normally black. The most obvious mistake seems to be on picnic baskets which PALM classifies as red. We believe these are included in Single-type questions due to the method used for constructing CoDa. To identify if an object has a single, multiple, or many typical colours, Paik et al. (2021) use a clustering algorithm together with manual assignment. However, the threshold of one-versus-many clusters can be hard to decide, and many objects would end up at the boundary. Also, depending on the number of annotators, the presented ground truth may be noisy when compared to the general population.

C Few-shot Prompts
One-shot. For one-shot, we prepend one randomly selected example from the dataset. The example is constructed by randomly selecting an object from the dataset and then choosing the colour with the highest probability answer from the ground truth. Some of the objects could have multiple reasonable colours (e.g., yellow will be chosen for bananas, even though they can be green or brown).