{'fraction_non_alphanumeric': 0.032681291412581505, 'fraction_numerical': 0.010902563290303846, 'mean_word_length': 4.603017305132377, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding and Mitigating the Effect of Outliers in Fair Ranking

Traditional ranking systems are expected to sort items in the order of their relevance and thereby maximize their utility. In fair ranking, utility is complemented with fairness as an optimization goal. Recent work on fair ranking focuses on developing algorithms to optimize for fairness, given position-based exposure. In contrast, we identify the potential of outliers in a ranking to influence exposure and thereby negatively impact fairness. An outlier in a list of items can alter the examination probabilities, which can lead to different distributions of attention, compared to position-based exposure. We formalize outlierness in a ranking, show that outliers are present in realistic datasets, and present the results of an eye-tracking study, showing that users scanning order and the exposure of items are influenced by the presence of outliers. We then introduce OMIT, a method for fair ranking in the presence of outliers. Given an outlier detection method, OMIT improves fair allocation of exposure by suppressing outliers in the top-k ranking. Using an academic search dataset, we show that outlierness optimization leads to a fairer policy that displays fewer outliers in the top-k, while maintaining a reasonable trade-off between fairness and utility.


INTRODUCTION
The primary goal of a ranker as used in a search engine or recommender system is to optimize the list in order to satisfy user needs by sorting items in their order of relevance to the query [22]. Recently there has been a growing concern about the unfairness towards minority groups caused by this simplistic assumption [5,6]. Several studies have proposed approaches to achieve fair ranking policies. The goal is to ensure that the protected groups receive a predefined share of visibility. Exposure-based methods [6,24,26,[31][32][33] quantify the expected amount of attention each individual or group of items receives from users, where attention is typically related to the item's position and based on the observation that users are more likely to click on items presented at higher positions [5,19].
But item position is not the only factor that affects exposure [13]. Inter-item dependencies also play a key role [8]. E.g., consider a user who is trying to buy a phone. When searching on an e-commerce platform, if an item in the list is on promotion and has a "Best Seller" badge, this can be distracting so that it gets more attention from the user, irrespective of its position in the list; the item would stand out even more if it is the only one with this feature.
We hypothesize that inter-item dependencies have an effect on examination probability and exposure of items. We focus on the case of having an outlier in the ranking and aim to understand and address its effect on user behavior. We hypothesize that exposure received by an item is influenced by the existence of an outlier in the list, and assume that this effect should be considered while allocating exposure to protected groups in a fair ranking approach.
We define outliers as items that observably deviate from the rest. The properties and method with which we identify outliers in a set of items are dependent on the task. The properties are observable item features that can be presentational in nature or correspond to ranking features used to produce the ranked list. E.g., in the e-commerce search example, if only one item in a result page has a "Best Seller" tag, it is an outlier based on this presentational feature.
To begin, we perform an exploratory analysis on the TREC Fair Ranking dataset. We observe that a large number of outliers exist in the rankings, where we use multiple outlier detection techniques to identify outliers based on the papers' citations as they can make an item more attractive and catchy than others.
Next, we perform an eye tracking study, where we measure the attention that each item in a ranked list gets through eye tracking, so as to show that users can actually perceive outliers in rankings.
We find that attention is more focused on outlier items. The scanning order and exposure received by each item may be influenced by the existence of outliers. Unlike other types of bias studied in search and recommendation [19,27,28,37,40], our eye-tracking study reveals that outlierness comes from inter-item dependencies. It affects the examination propensities for items around the outlier in a way that is less dependent on the position and based on relationships between items presented together. However, it is translated to bolded keyword matches in the title and abstracts, which can be calculated for each item separately, independent of its neighbors. Attractiveness does not alter the examination model based on position bias, and only results in relatively more clicks on items when they are presented with more bolded matched keywords [40].
While allocating fair exposure to protected items or groups in a fair ranking solution, we should account for the effect of outliers. We propose an approach to account for the existence of outliers in rankings without damaging the utility or fairness of the ranking by mitigating outlierness, called OMIT. OMIT jointly optimizes (1) user utility, (2) item fairness, and (3) fewer outliers in top-positions as a convex optimization problem that can be solved optimally through linear programming. Via its solution, we derive a stochastic ranking policy using Birkhoff-von Neumann (BvN) decomposition [7]. OMIT reduces the number of outliers at top-10 positions on the TREC 2020 dataset by 80.66%, while maintaining the NDCG@10, compared to a state-of-the-art ranking baseline.
The main contributions of this work are as follows: (1) we introduce, study and formalize the problem of outlierness in ranking and its effects on exposure distribution and fairness; (2) we run an extensive eye-tracking user study in two search domains to support our hypothesis about the existence of an effect of outliers on items' exposure; (3) inspired by our analysis, we propose OMIT, an efficient approach that mitigates the outlierness effect on fairness; (4) we compare OMIT to competitive baselines on two TREC datasets in terms of fairness, outlierness, and utility; OMIT is able to remove outliers while balancing utility and fairness; and (5) we make the data from our eye-tracking study plus the code that implements our baselines and OMIT publicly available.

BACKGROUND
Exposure and utility. Consider a single query , that we will often leave out for notational simplicity, for which we want to rank a set of documents D = { 1 , 2 , . . . , }. Suppose we are given document utilities u ∈ R , where is a proxy for the relevance of document for . Let v ∈ R , be the attention vector, where denotes how much attention a document gets at position , and which is decreasing with the position. This vector encodes the assumed position bias, e.g., = 1/log(1 + ).
We require a probabilistic ranking in the form of a doubly stochastic document-position matrix P ∈ [0, 1] × where denotes the probability of putting document at position . Such a matrix can be decomposed into a convex combination of permutation matrices, which allows us to sample a concrete ranking [32].
The exposure of a document under ranking P denotes the expected attention that this document will get. Using the position based attention vector , this can be modeled as a function of the ranking and position bias: Exposure( |P) = =1 . The expected utility of a ranking P is the sum of the documents' utilities weighted by the exposure given to them by P: Without fairness considerations, a utility-maximizing ranking can be found by sorting the documents in descending order of utility.
Group fairness. Suppose now that the documents D can be partitioned into two disjoint sets D and D , where documents in D belong to a historically disadvantaged group (e.g., publications from not so well established institutes), and those in D belong to the privileged group (e.g., publications from well-established institutes). We want to ensure a certain notion of fairness in the ranking. We want to avoid disparate treatment of the different groups. We use the disparate treatment ratio [32], which measures how unequal the exposure given to the disadvantaged group (in relation to the corresponding utility of the disadvantaged group) is compared to the corresponding ratio of the privileged group, as: Note that dTR is 1 if the groups are treated fairly and smaller than 1 if the ranking is unfair towards the disadvantaged group. We often encounter disparate treatment when only optimizing for the expected utility of a ranking [5,6,32]. We can find a utility maximizing ranking P that avoids disparate treatment by solving the following optimization problem [32]: such that 1 ⊤ P = 1 ⊤ (row stochasticity) where 1 denotes a vector and f is the vector constructed to encode the avoidance of disparate treatment, with where 1 ∈D = 1 if document is in the disadvantaged group and 0 otherwise (and analogously for 1 ∈D ) [32].
Degrees of outlierness. Outliers are items that deviate from the rest of the data [18]. They can be interesting observations or suspicious anomalies. Either way, they are considered noise that can affect the statistical analysis. We describe three outlier detection methods that we will use later in the paper. Let = { 1 , . . . , | ∈ R} be a set of values for which we want to identify outliers.
Median Absolute Deviation (MAD). Although it is common practice to use Z-scores to identify possible outliers, this can be misleading (particularly for small sample sizes) due to the fact that the maximum Z-score is at most ( − 1)/ √ . Iglewicz and Hoaglin [17] recommend using the modified Z-score: = 0.6745( −˜)/MAD, where MAD is the median absolute deviation and˜is the median of . These authors recommend that modified Z-scores with an absolute value of greater than 3.5 be labeled as potential outliers.
Median K-Nearest Neighbor (MedKNN). This model [3] uses the K-Nearest Neighbor algorithm to define a distance-based outlier detection method. For each point we have a value ( ) as the weight calculated from the nearest neighbors; outliers are the points with the largest values of . We use Median K-Nearest Neighbor, which computes ( ) as the median distance of to its neighbors. To find the nearest neighbors, the method linearizes the search space and uses Hilbert space-filling curve to search efficiently; the method scales linearly in the dimensionality and the size of the data.
Copula-Based Outlier Detection (COPOD). COPOD [21] is a novel outlier detection method based on estimating tail probabilities using empirical copula. COPOD uses empirical cumulative distribution functions (ECDFs) to compute tail probabilities. These tail probabilities estimate the probability of observing a point at least as extreme as for each data point. If is an outlier, the probability of observing a point as extreme as should be small. It means that this point has a rare occurrence. This method is deterministic and efficient, and scalable to high dimensionality data.

OUTLIERS IN RANKING
A common assumption is that exposure is a function of position [6,24,26,32,36]. We argue that this assumption holds only if ranked items can be deemed similar, meaning that no item is perceived as an outlier. Below we introduce outliers in the context of ranking. We then determine that outliers are present in rankings in realistic datasets. We also report on an eye-tracking study that shows that the presence of outliers in ranked lists impacts user behavior.
A definition of outliers in ranking. For a ranked list, we define outliers as items that stand out among the window of items that the user can see at once, drawing the user's attention. Outlier items often have (visible) characteristics that distinguish them from their neighbors. E.g., consider Fig. 1, which shows a result page for the query "smart phone". The result page view consists of 6 products, each presented with characteristics such as title, image, and price. The item at position three deviates from the other items in terms of several visual characteristics; it has more details, some promotive tags, and bold keywords. Other features, such as more positive reviews, or a higher price, may also influence the user's perceived relevancy. In this example, the third item can be considered as an outlier according to such visual characteristics.
Formally, we define outliers in ranking as follows. Consider a ranked list of items in D = { 1 , 2 , . . . , }, that has been produced in response to a query. We call an observable characteristic of an item in a ranked list an observable item feature. These features can be purely presentational in nature, like the bold keywords in Fig. 1, or correspond to ranking features used by the search engine to produce the ranked list, e.g., the average user rating.    Note that detecting an item as an outlier in a ranking depends on the context in which we see the item. Throughout the paper, we consider the full ranked list of items as the context in which we detect outliers. In Section 6 we study varying sizes for the context.
Moreover, it is possible to use multiple observable features to detect the outliers. For example, we can consider image size as 1 and price as 2 , and then use any combination of these two feature values to present item .
Below, when we refer to an item being an outlier in a given ranked list, we assume that it is clear from the context what outlier detection method M and observable item feature are being used.
Do outliers in ranking exist? To determine whether outliers are present in rankings in datasets, we take a retrieval test collection, compute feature values for one of the (potentially observable) rankings features appropriate for the collection, and determine whether there are outliers among the top-20 documents returned for the test queries (using ListNet as the ranker, see Section 5). For the experiments in Section 6 we use the academic search dataset provided by the TREC Fair Ranking track. 1 It contains information about papers and authors extracted from the Semantic Scholar Open Corpus. 2 It comes with queries and relevance judgments; see Table 1 for some descriptive statistics.
We used the number of citations as observable feature for this dataset as they can make an item more attractive than others (when reported). In the remainder of the section, we report the analysis only on the TREC 2020 data, as we observed similar trends in both datasets. Fig. 2 shows the mean, maximum, and minimum of papers' citation counts for all search sessions in the data. There is a high variance between mean and maximum citations, which implies that the data is outlier-prone based on this feature. We plot outlier counts for each position in the top-20. Fig. 3 depicts the number of relevant and non-relevant outliers detected by the outlier detection methods introduced in Section 2 at different positions. The stacked bars show that in spite of the attractiveness of outliers, most of these items are irrelevant, judging by the click data. In total, 88.5%, 89.8%, and 90.1% of the outliers are irrelevant when MedKNN, COPOD, and MAD are employed as the outlier detection method, respectively. The average percentage of irrelevant documents in the top 20 positions in the dataset is 83.3%. This suggests that by pushing these items to lower positions we can improve the degree of outlierness without jeopardizing utility.
Do users perceive outliers in the ranking? Outliers are present in realistic datasets, but do they impact user behavior? Prior studies stress the importance of relationships between ranked items [30], but it is unknown how an outlier in a ranking affects the examination probability. To address this gap, we conduct an eye-tracking study. We ask participants to interact with search engine result pages, as they normally would, and find the items that they prefer and think are relevant. We track their eye movements via an online webcam-based eye tracking service. 3 We use two scenarios, e-commerce, and scholarly search. We focus on a list view; in both scenarios, participants are able to see all items in one page. For each scenario, we include two result pages, one without an outlier item and one with (as in Fig. 1). In the absence of outliers we expect participants to follow the position bias examination assumption [19]; in the presence of outliers, we expect that users' attention is diverted towards them.
We recruit 40 university students and staff for both scenarios. In the instructions, we describe the overall goal of the research and ask participants to read the instructions carefully. We describe what webcam-based eye tracking is and that the eye-tracking service will ask them for access to their webcam. We instruct participants to first read and understand the query and then start scanning the result page as if they submitted the query themselves.
For reporting, we consider four eye-tracking measures based on participants' eye fixations: (1) fixation count (the number of fixations within an area; more fixation means more visual attention); (2) time spent (shows the amount of time that participants spent on average looking at an area); (3) Time To First Fixation (TTFF; the amount of time that it takes participants on average to look at one 3 RealEye, https://realeye.io.  area for the first time); and (4) revisit count (indicates how many times on average participants looked back at the area) [15].
Outliers in e-commerce. For this scenario, we mimick the Amazon Marketplace search result page. Fig. 1 depicts our example ranked list with an outlier. The third item in the list stands out from other items for different reasons, including price and sales-related tags (e.g., being on sale), as well as other information that is available for this item. Comparing Fig. 4a and 4b, we see that in the presence of an outlier, items at the top of the list receive less attention, contradicting the position bias assumption. Fig. 5(a) reports the eye-tracking measures for the e-commerce scenario. We highlight the outlier in each list with an asterisk. In the no-outlier condition, participants exhibit linear behavior in terms of scanning the items. The highest number of fixations, time spent, revisits belongs to the items on the top of the list, and it decreases as we go down the list. TTFF demonstrates a linear behavior of the first fixation time, that is, most of the participants started scanning the results from top to bottom. The ranked list with an outlier exhibits very different measurements. Attention is more focused on the outlier item. Also, we see that from the TTFF values, the average time for the first fixation is the lowest for this item, suggesting that the scanning order and exposure are influenced by the existence of the outlier. This is also evident by comparing the heatmaps in Fig. 4a and 4b.
Outliers in scholarly search. In the second scenario, we study the  effect of outliers on scholarly search result pages. To this end, we mimick the result page from PapersWithCode. 4 To save space, we omit the eye fixation heatmap for the result page without outliers; it shows the familiar F-shape. Fig. 4c shows the eye fixation heatmap for the result page with an outlier item; the fourth item has a different thumbnail and a large number of GitHub stars, making this item stand out in the list. Similar to the e-commerce scenario, the eye fixations are very different from the F-shape typical for the no-outlier case. Fixations and time spent are the highest for the outlier item, suggesting that it draws lots of attention, and contradicts the cascade examination assumption.
However, different from the e-commerce case, we do not observe as big a difference in the TTFF values between the conditions with and without outliers, suggesting that the order of item scans is not affected as much as in the e-commerce example.

MITIGATING OUTLIERNESS IN FAIR LEARNING TO RANK
We now present a ranking algorithm for mitigating outlierness for fairness in ranking, called OMIT, that simultaneously accounts for item fairness and outlierness effect requirements. Based on the observations reported in the previous section we know that outliers can influence exposure and examination order, in a way that can be considered as a type of bias. We take a first step towards mitigating the outlierness phenomenon by proposing a remarkably simple, yet effective solution that removes outliers from the top positions where the distribution of exposure is most critical. Our solution aims at decreasing outlierness in the top-positions, while retaining the ranking's utility and fairness with position-based assumptions.
OMIT is based on the linear programming method described in Section 2. In addition to optimizing for user utility while staying within the fairness constraints, our goal is to reduce the number of outliers in the top-of all rankings. OMIT has two steps. In the first, we search for the marginal rank probability matrix that satisfies item group fairness by solving a linear program that optimizes both for user utility and fewer outliers in the top-items with linear constraints for fairness. In the second, we derive a stochastic ranking policy from the solution of the linear programming method 4 https://paperswithcode.com using the Birkhoff-von Neumann decomposition [7]; cf. also [32].
Step 1: Computing MRP matrix. Let D = { 1 , . . . , } be a set of items that we aim to rank for a given query . Each ranking (·| ) corresponds to some permutation matrix that re-orders the elements of D . 5 As discussed in Section 3, to determine which items are outliers, we use the domain specific observable item features ( 1 ), . . . , ( ) that potentially impact the user's perception of an item. Using these characteristics as features, one can use any outlier detection method to determine which items should be considered outliers. The majority of outlier detection methods, including the ones we use (Section 3), find outliers by calculating scores that indicate the degree of outlierness. This results in a list of outlierness values M( ( 1 )|D), . . . , M( ( )|D), where ≤ is the size of the outlierness context that the algorithm takes into account while detecting the outliers. We define a vector o D , that contains, for each document, the information whether it is an outlier with respect to the full ranked list: We use o and o D interchangeably. Note that we are considering outliers in the context of the full list, = , i.e., the outlier detection algorithm takes the whole ranked list as input. We assume that items that are perceived as outliers in this context are likely to be perceived as outliers when seen in the smaller context of the topitems. Below, we show that this heuristic works well in practice. OMIT works by pushing outliers away from the top-. Let be the permutation matrix corresponding to a ranking . The amount of outlierness that a ranking places in the top-are equal to where h = (1, . . . , 1, 0, . . . , 0) is a vector containing 1 for the first positions and 0 for all positions after that. Similarly, the expected outlierness, that is placed in the top-by P is given by While optimizing for user utility we can use the expected outlierness 5 For simplicity, we interchangeably use ( ·) and ( · | ), as well as D and D .

4:
← BvN-Decomposition(P) 5: Return to add an objective that will function as a regularization term, penalizing ranking policies with outliers in the top-. We extend the linear programming approach described in Section 2 to solve: For item fairness, we adopt the disparate treatment constraints as described in Section 2. Both terms of the optimization objective, and the constraints for fairness and finding a doubly stochastic matrix are linear in 2 variables. Hence, the resulting linear program can be solved efficiently using standard optimization algorithms [32].
Step 2: Constructing a stochastic ranking policy. The solution to the linear program P is a matrix indicating the probability of showing each item at any position. To generate actual rankings, we need to derive a stochastic ranking policy from P and sample rankings to present to users. We follow [32] and use Birkhoff-von Neumann decomposition to compute , which decomposes the doubly stochastic matrix P into the convex sum of permutation matrices = 1 1 + · · · + , with 0 ≤ ≤ 1, = 1 [7]. This results in at most ( − 1) 2 + 1 rankings [23], corresponding to , that are shown to the user with probability , respectively. 6 OMIT model summary. Algorithm 1 presents an overview of OMIT. OMIT takes as input the initial ranking (optimized for utility), outlier detection method M, outlierness context size , and the number of top items that we aim to remove outliers . In line 1, o D is created for a given outlier detection technique and outlierness context size, followed by line 2 where we create the h list that takes into account the top of the list that we aim to mitigate outlierness. In line 3, we solve the linear program that jointly solves the fairness and outlierness problem and pass the stochastic ranking in line 4 to the BvN decomposition algorithm. Finally, we return the output of the BvN method as the output.

EXPERIMENTAL SETUP
We target the following research questions: (RQ1) How do different outlier detection methods affect OMIT's performance? (RQ2) How does our OMIT trade-off between utility, fairness, and outlierness, compared to baselines? (RQ3) We adopt the constraints proposed in [32] (called FOE) to optimize a ranked list for fairness and utility through linear programming, as described in Section 4. Given that OMIT adds additional constraints, is the overall linear program more effective when we treat the doubly stochastic matrix constraints as hard or soft constraints? (RQ4) How does changing the context of detecting outliers affect OMIT's outlierness improvement and utility? (RQ5) How does changing affect OMIT's outlierness improvement and utility in the top-positions?
Data. We use data from the TREC Fair Ranking 2019 and 2020 track (see Section 3). We make the group definitions over the two datasets consistent. As for the TREC 2019 data, we bin the original article groups into two classes. For the TREC 2020 data, we adopt the group definitions from the original data, that is, documents are assigned to two groups based on their authors' h-index. Moreover, we follow the TREC setup to generate query sequences, leading to multiple occurrences of the same query, using the provided frequencies. Specifically, we evaluate on a query sequence of size 10, 000, including all the queries in the evaluation data.
Evaluation metrics. We evaluate methods for fair learning to rank in the presence of outliers in terms of utility, item fairness, and outlierness. For utility and fairness, we use NDCG and dTR (see Eq. 2), respectively, as our metrics and report their expected values. To measure the expected outlierness of the policy P up to position in the ranking, we use Outlierness D (P| M) as defined in Eq. 7. Similarly we define the expected number of outliers up to position in ranking for policy P as where o = 1 >0 (o ) is the binarized version of o where each outlier item is assigned 1, and all the rest are assigned 0.
Compared methods. To evaluate OMIT, we build several baseline methods, combining different options for each component of our model (initial ranking, fairness of exposure, outlier mitigation): • Initial ranking: The initial ranking of all compared methods is generated using ListNet [9]. ListNet is a learning to rank model, optimizing for utility. We use it in our experiments to create initial ranked lists, D , using the click data provided in the training set, with 30 maximum epoch, and a validation ratio of 0.3. • Fairness of exposure: We use two variants of FOE [32] based on hard vs. soft doubly stochastic matrix constraints, and call them FOE and FOE , respectively. 7 • Outlier mitigation: We employ two outlier mitigation techniques, namely, RO and OMIT. RO removes all the outlier items detected by M from the ranking, while OMIT is our proposed outlier mitigation method as described in Section 4. We specify methods as combinations of the three components mentioned above. E.g., "ListNet + FOE + OMIT" uses the initial ranked list produced by ListNet, applying FOE fairness post-processing with hard constraints and the OMIT outlier mitigation model.

EMPIRICAL RESULTS
Effect of outlier detection method. We address RQ1 by changing the outlier detection method, while keeping the other parts of the model fixed. Table 2 reports the results of using three different outlier detection methods in OMIT. For comparison, we also report the results of ListNet without outlier mitigation (row a) and report the relative improvements. All three outlier detection methods effectively reduce outlierness compared to the baselines. COPOD achieves the best results by reducing outlierness by 80.3% and 80.6% in terms of Outlierness@10 on the TREC 2019 and 2020 data, respectively. In terms of dTR, COPOD outperforms MAD and MedKNN Table 2: Comparing loss in fairness and utility, with gains in outlierness for different outlier detection methods on the TREC 2019 and 2020 Fair Ranking data. Models used: (a) ListNet and (b) ListNet + FOE + OMIT. Δ values denote the percentage of relative improvement compared to (a). * refers to statistically significant improvements compared to (a) using a two-tailed paired t-test ( < 0.05). on both datasets where it increases dTR by 4.5% on TREC 2020. We see no significant difference in the utility achieved by the methods. Given that COPOD is parameter-free and scalable, we prefer it over the other two methods. For the remaining experiments, we choose COPOD as the outlier detection method.
Utility, fairness, and outlierness trade-offs. To answer RQ2, we turn to Table 2, which shows the results for OMIT and the baseline methods in terms of utility, fairness, and outlierness. Although ListNet is purely optimized for utility, it does not achieve the highest NDCG in all cases. This suggests that optimizing for fairness and outlierness could even improve the utility. As shown in Fig. 3 there is a high density of outliers among top positions that are mostly irrelevant. Therefore, when OMIT pushes these items to lower positions, it improves outlierness and utility measures simultaneously. Moreover, we see that mitigating outlierness does not cause any significant effect on dTR, showing that OMIT is capable of retaining position-based item fairness.  2020 data, respectively. We return the original ranking as the output when FOE does not find an optimum solution. To fix this problem we implemented the constraint for doubly stochastic matrix as a soft constraint and we check for validity of the permutation matrices generated by the decomposition algorithm. Table 3 demonstrates the effectiveness of the soft variant of FOE (row g vs. f).
Effect of . To answer RQ4, we examine the effect of the parameter, which determines the outlierness context size. We change from 10 to 40 items while keeping other parameters fixed, and observing the behavior of the model. This is important since the outlierness of an item depends on its context, e.g., an item can be considered as an outlier in the top-10 items, but may not be an outlier in the top-20 items. The two left plots in Fig. 6 show the outlierness improvements (compared to ListNet) and utility in terms of NDCG@10 for different values of on the TREC 2019 and 2020 data. We see that Outlierness@10 improves for larger values of , suggesting that determining outlierness of items in a bigger pool of items is more accurate and allows OMIT to mitigate the outliers in the top-10 positions more effectively.

Effect of .
To answer RQ5, we study the effect of changing when optimizing for mitigating outlierness in top-positions. We are interested in finding out how utility and outlierness are influenced when optimizing for mitigating outlierness for a longer list of top ranks ranging from 10 to 40. This mimics the cases where more items can be shown to a user, hence outliers in longer lists can be observed by the user. The left plots in Fig. 6 depict the results for both datasets. We observe that outlierness improvement drops for greater values of since it is more challenging for the algorithm to push all outliers to lower positions. Fig. 3 shows that most outliers are located at top positions, so greater values of do not necessarily translate to more outliers. The changes in utility scores of the lists are marginal, with a 0.5% increase and 1.6% decrease for larger  values of for the TREC 2019 and 2020 data, respectively. The difference in utility of the two datasets is due to the fact that there are more relevant outliers in TREC 2019 than in TREC 2020.

RELATED WORK
Bias in implicit feedback. Users' implicit feedback, such as clicks, can be a great source of relevance judgment that has been shown to help improve search quality [2]. clicks may be misleading due to different types of bias, which causes the probability of a click to differ from the probability of relevance. Recent work on discovering and correcting for different types of bias in logged click data concerns position bias [19,20], presentation bias [40], selection bias [29], trust bias [2,35], popularity bias [1], and recency bias [12]. Inter-item dependencies affect the perceived relevance of items [13]. We introduce a phenomenon that is anchored in inter-item dependencies and may result in biased clicks. Our work differs from previously discussed types of bias by considering interitem relationships. Showing items as outliers can make them more attractive to users, influencing their perceived relevance. Presentation bias [40] concerns a related effect; bold keywords in titles make some items appear more attractive. However, this definition of attractiveness is independent of other items in the list. Metrikov et al. [25] show that click-through rates can be manipulated by adding more images to top positions next to the ad slots on the search result page; they did not study the effect on item exposure or biased clicks. We focus on the effect of outliers, as an inter-item dependency on the examination probability and item exposure, and introduce it as a potential source of bias in click data.
Fair ranking. Following [42], we distinguish two ways of measuring fairness of rankings. Work on probability-based fairness determines the probability that a ranking is the result of a fair process [4,10,11,16,34,39,41]. Exposure-based methods determine the expected exposure for each item in the ranking and aim to ensure that this exposure is fairly distributed [6,14,24,26,[31][32][33]38]. Our work belongs to the second category. To estimate the expected exposure of each item or group, we need to take into account different types of bias that the user might have when observing system outputs. Previous work has mostly focused on position bias [6,32,38]. We emphasize the role of inter-item dependencies in the exposure that an item receives, which can be a source of unfairness when not considered in computing the expected exposure. We extend the re-ranking approach introduced in [32], to not only produce fair rankings but also avoid showing the outliers in the top-.
An important effort for developing a benchmark for evaluating retrieval systems in terms of fairness is the TREC Fair Ranking track (see footnote 1). We expand the use of the track's resources to include the study of outliers in fair ranking.

CONCLUSION & FUTURE WORK
We introduced and studied a phenomenon related to fair ranking, that is, outlierness. We analyzed data from the TREC Fair Ranking track and found a significant number of outliers in the rankings. We hypothesized that the presence of outliers in a ranking impacts the way users scan the result page. We confirmed this hypothesis with an eye-tracking study in two scenarios: e-commerce and scholarly search. We proposed OMIT, an approach to mitigate the existence and effect of outliers in a ranked list. With OMIT, we introduced a ranking constraint based on the outlierness of items in a ranking and combined it with fairness constraints. We formulated the problem of outlier mitigation as a linear programming problem and produced stochastic rankings, given an initial ranking. Using OMIT one can reduce outliers in rankings without compromising user utility or position-based item fairness. We analyzed the effects of different outlier detection approaches and compared their results. Our experiments also showed that there is a trade-off between the depth of outlier detection and user utility. Now that we have established the impact of outliers in rankings, future work on fair ranking should consider the presence of outliers by default.
One limitation of our work is our focus on removing the bold outliers defined in the context of the whole list from the toppositions. We plan to improve our model to mitigate the outlierness of all sliding windows of size . In addition, we want to improve the performance for cases where outliers are relevant items, e.g., by considering alternative methods of outlier mitigation. Finally, a natural extension of our work could be to quantify the effect that an outlier in the presentation of the ranking can have on the examination probability distribution; this could open the door to unbiased learning to rank approaches that counter the outlier bias in logged user data.

DATA AND CODE
To facilitate reproducibility of our work, all code and parameters are shared at https://github.com/arezooSarvi/OMIT_Fair_ranking.