{'fraction_non_alphanumeric': 0.0314410480349345, 'fraction_numerical': 0.00944819372766971, 'mean_word_length': 4.969668246445497, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Toward reliable measurements of perceptual scales in multiple contexts

A central question in psychophysical research is how perceptual differences between stimuli translate into physical differences and vice versa. Characterizing such a psychophysical scale would reveal how a stimulus is converted into a perceptual event, particularly under changes in viewing conditions (e.g., illumination). Various methods exist to derive perceptual scales, but in practice, scale estimation is often bypassed by assessing appearance matches. Matches, however, only reflect the underlying perceptual scales but do not reveal them directly. Two recently developed methods, MLDS (Maximum Likelihood Difference Scaling) and MLCM (Maximum Likelihood Conjoint Measurement), promise to reliably estimate perceptual scales. Here we compared both methods in their ability to estimate perceptual scales across context changes in the domain of lightness perception. In simulations, we adopted a lightness constant, a contrast, and a luminance-based observer model to generate differential patterns of perceptual scales. MLCM correctly recovered all models. MLDS correctly recovered only the lightness constant observer model. We also empirically probed both methods with two types of stimuli: (a) variegated checkerboards that support lightness constancy and (b) center-surround stimuli that do not support lightness constancy. Consistent with the simulations, MLDS and MLCM provided similar scale estimates in the first case and divergent estimates in the second. In addition, scales from MLCM–and not from MLDS–accurately predicted asymmetric matches for both types of stimuli. Taking experimental and simulation results together, MLCM seems more apt to provide a valid estimate of the perceptual scales underlying judgments of lightness across viewing conditions.

The statistical models in MLDS and MLCM take the stochasticity of observers' judgments into account. Their parametrizations place an additive and Gaussian noise source at the decision stage.
In MLDS the stimulus triad x 1 , x 2 and x 3 evokes a deterministic perceptual response Ψ(x 1 ), Ψ(x 2 ), Ψ(x 3 ) and the observer compares the two perceptual intervals with a differencing rule expressed in the decision variable ∆: where Ψ i (x) is the perceptual scale in the i-th context, ∼ N (0, σ 2 ), and σ 2 is the decision noise variance. The observer responds that the pair (x 2 , x 3 ) is perceived as having the biggest perceptual difference if ∆ MLDS > 0, (x 1 , x 2 ) otherwise.
Similarly, for MLCM we consider all possible paired comparisons of context and luminance. Thus we have that the decision variable is where the observer compares the luminances x 1 and x 2 in contexts i and j, respectively, and ∼ N (0, σ 2 ). The observer responds that x 2 is perceived as lighter if ∆ MLCM > 0, x 1 otherwise.
Using a binomial generalized linear model (GLM), MLDS and MLCM find the values of Ψ(x) that maximize the likelihood given the observers' responses. Importantly, they also provide an estimate of the decision noise parameter,σ.
We can parametrize the model differently by shifting the noise from the decision stage to the perceptual stage. This parametrization is motivated by the potential comparison of this class of scaling methods with performance-based methods, as they instantiate the same assumptions taken in the simplest version of signal detection theory (equal-variance, independent, Gaussian distributed variables), and thus perceptual scales can be expressed in "d' units" (see applications in e.g. Devinck & Knoblauch, 2012;Aguilar, Wichmann, & Maertens, 2017).
We reparametrize the models by assuming that the noise comes only from the internal dimension Ψ(x) and that there is no decision noise. Each perceptual response is an equal variance, Gaussian distributed random variable with mean equal to the deterministic Ψ(x), where σ 2 P is the noise variance at the perceptual level. Then we can rewrite the decision variables for MLDS and MLCM as follows MLDS and MLCM provide a noise estimateσ that reflects the noise of the entire decision model. The next step is to expressσ as a function of the perceptual noise σ 2 P . The decision variables are a linear combination of Gaussian random variables. The variance always adds when linearly combining Gaussian random variables. Thus, the variance of the decision variables isσ as in the decision variable for MLDS there are four terms, and in the decision variable for MLCM only two terms. Assuming that MLCM and MLDS probe the same internal dimension Ψ(x), we can now put these last two equations together, and we obtained the noise relationship between MLCM and MLDŜ

Goodness of fit
Evaluating the goodness-of-fit (GoF) of general linear models (GLMs) with binomial responses (as in MLDS and MLCM) is not as straightforward as with Gaussian data. We followed the approach suggested by Knoblauch and Maloney (2012) and Wood (2006) to analyze the appropriateness of our scale estimates. The procedures are implemented in the MLDS and MLCM R packages (Knoblauch & Maloney, 2012). Using bootstrapping we generated responses from the empirically estimated scales. We then fit scales to these simulated data and calculated the deviance residuals for each of the bootstrapped datasets. Finally we compared the actual deviance residuals to the distribution of the simulated residuals. There are two ways to check goodness of fit.
In the first comparison, the cumulative distribution of empirically observed residuals is plotted together with a 95 % confidence envelope which is obtained from the simulated residuals (Fig. S1A). If our distributional assumption is correct, then the envelope represents the residual distribution. For a satisfactory goodness of fit the empirical scale should fall inside the envelope. We quantified this GoF test by calculating the percentage of trials (P in ) in which the empirical residuals were located inside the envelopes. Additionally, the plotting routine detects and visualizes individual trials with high residuals which may point to outliers (see below). The second comparison tests for systematic patterns in the residuals. In a simple linear model (y = ax+b+ , ∼ N (0, σ 2 )) the residuals (y−ŷ) are plotted against the predicted values (ŷ) to check that the residuals scatter randomly around zero. Analogously, for binomial GLMs we sort the residuals according to the fitted values and check for sequences of only positive or only negative deviance residuals. Practically we calculate the number of times that adjacent residuals cross zero. To put the empirically observed number of zero-crossings into perspective we compare it with the distribution of zero-crossings in the simulated data (Fig. S1B). The observed value should be within the probability mass of the simulated distribution, i.e. their p-value is not less than 0.01, indicating a random distribution of residuals and hence a satisfactory goodness-of-fit.
On a first evaluation we found that 91 % (29 out of 32) of the cases scales in variegated checkerboards had an satisfactory goodness of fit, against 41 % (13 out of 32) of the cases for the center-surround stimulus (see Suppl. Tables S1 and S2 for details). As many scales did not have a satisfactory goodness of fit, we employed an outlier removal procedure. The procedure is described in Knoblauch and Maloney (2012, pp. 219-222) and it consists of removing the trials which are flagged as having a deviance residual higher than an arbitrary threshold (set to ±2 in our case). The trials flagged as 'outliers' can be seen in the first comparison plot (Fig. S1A) as datapoints with x-values to the extreme left or right, corresponding to a deviance residual outside the range [−2, 2].
We identified outlier trials in all scaling data using the same criterion as described above, we removed them from their respective datasets, and refitted the models (MLDS or MLCM) to find new scale estimates. These newly estimated scales passed the goodness of fit test in 100 % of the cases for variegated checkerboards, and in 94 % (30 out of 32) of the cases for center-surround stimuli (scales for observers O4 and O5 in 'plain view' did not pass). These updated scales are the ones reported in all results in the main text.

Independent, additive and saturated models in MLCM
MLCM includes three different statistical models: the independent, the additive, and the saturated model. The independent model considers that perceptual judgments can be explained by only one of the stimulus dimensions. The additive model considers that judgments can be explained by a sum of the effects from each stimulus dimension. And finally the saturated model is the most general and it considers the contribution of each stimulus combination presented to the observer, thus allowing interactions that are more complex than a sum among stimulus dimensions. The model is 'saturated' as it has the maximum number of possible parameters. These three models are nested, being the saturated the most general, followed by the additive and finally the independent model. Model selection among these three options is done using the nested likelihood ratio test. More details can be found in Knoblauch and Maloney (2012) and in Gerardin, Devinck, Dojat, and Knoblauch (2014).
For our current use of MLCM we considered all three models. As a reminder, the two stimulus dimensions in our study were luminance and a categorical dimension for viewing context (plain, dark transparency, light transparency). And we evaluated three observer models: lightness constant, luminance-based and contrast-based models.
MLCM with the independent model is appropriate only for the luminance-based observer, as judgments would only depend on one stimulus dimension, Ψ(x) i.e. luminance. It is however not appropriate for any other observer model or any other case that we can consider. Thus, we evaluated the next more general model, the additive model.
The additive model would be appropriate if the judgments would depend on the luminance dimension plus an additive offset due to context. An additive offset is however not enough in capturing the interaction of the luminance dimension with the context dimension. In fact, for the lightness constant observer, luminance is transformed to perceived lightness by a multiplicative factor. This multiplicative factor cannot be modeled with additive conjoint measurement (see Fig. S2 for an illustration). Consequently we used MLCM with the saturated model. In this way we cover all scenarios using the most general model.

Estimation issues in MLCM
During our use of MLCM we encountered technical issues worth of mentioning. The general linear model (GLM) -the model underlying the scales' estimation in MLCM and MLDS -often outputs a 'complete separation' warning, which indicate that the model is ill-constrained, having parameters that completely determine the outcome variable. This is not an uncommon occurance in binomial regression (Kosmidis & Firth, 2009). In both MLCM and MLDS the problem can arise when measured proportions of observers judgments are either all 0% or 100%. That could occur when the stimulus spacing is too coarse relative to the observer's internal noise, and thus all judgments are too easy for the observer.
In the present study the 'complete separation' problem did not affect the point estimates of the scales, but it became problematic when calculating confidence intervals using bootstrap. It led to unstable bootstrap estimates and a multi-modal bootstrap distribution, which in turn led to highly skewed confidence intervals. We overcame the problem by setting a coarser stopping rule in the optimization algorithm underlying the GLM fitting routine, in this way avoiding the multi-modality. Alternatively other GLM fitting methods that tackle specifically the problem of 'complete separation' could be used, for example the bias reduction methods implemented in the package brglm2 (Kosmidis, Pagui, & Sartori, 2018).  Table S1: Goodness of fit measures for scales in variegated checkerboards before and after outlier removal. Pin: percentage of residuals inside envelope (see Fig. S1A), p: p-value statistic of zero-crossings distribution (see Fig. S1B). Asterisks mark the cases with inappropriate goodness of fit. Detailed description can be found in the text.   Figure S4: Individual observer data. Consistency between matching data and prediction from MLCM perceptual scales in variegated checkerboards, for each observer individually. Datapoints represent mean ± 95 % C.I.