{'fraction_non_alphanumeric': 0.03495058902350511, 'fraction_numerical': 0.015800346155993523, 'mean_word_length': 4.771387143547607, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Overview of the SV-Ident 2022 Shared Task on Survey Variable Identification in Social Science Publications

In this paper, we provide an overview of the SV-Ident shared task as part of the 3rd Workshop on Scholarly Document Processing (SDP) at COLING 2022. In the shared task, participants were provided with a sentence and a vocabulary of variables, and asked to identify which variables, if any, are mentioned in individual sentences from scholarly documents in full text. Two teams made a total of 9 submissions to the shared task leaderboard. While none of the teams improve on the baseline systems, we still draw insights from their submissions. Furthermore, we provide a detailed evaluation. Data and baselines for our shared task are freely available at https://github.com/vadis-project/sv-ident


Introduction
Social science publications often use and reference survey datasets, containing hundreds or thousands of questions, using so-called survey variables. 1 While publications may focus only on a specific subset of these variables, explicit references are usually missing: the lack of explicit links between survey variables and publications, in turn, limits access to research along the FAIR principles (Wilkinson et al., 2016). To address this issue, we propose a task where variable mentions in unstructured documents are linked to items from a catalog of survey research datasets using Natural Language Processing (NLP) methods. Automatically identifying which variable is mentioned in a given text is challenging due to the diverse linguistic realizations of variables (Zielinski and Mutschke, 2018). A short example text is shown in Figure 1. All three In the subsequent section, we focus on two theoretical concepts -mere exposure and hostile media perceptions -both of which have not been employed within the context of European identity.
The dependent variable (identity) was operationalized by two indicators: Citizens were asked to answer the following question: "Please tell me how attached you feel to the European Union" by using a scale from 1 (not attached at all) to 4 (very attached).
Furthermore, the item "You feel you are a citizen of the EU" was used in index building.
... sentences mention and are linked to relevant variables. The first sentence mentions three concepts: 2 mere-exposure effect, hostile media perceptions, and European identity. The first two concepts are defined later in the text (we omit their links in this example), while the latter is defined in the bottom two sentences in the figure. The second and third sentences both are explicit mentions, as they include direct quotations of variable questions. Ideally, a system should link relevant variables to each of the sentences in the example. Specifically, when only provided the given context, it should link the first sentence to the variables QD2_3 and QD3_1, the second sentence to QD2_3, and the third sentence to QD3_1. A larger variant of the example is provided in Figure 3 in the Appendix.
The Survey Variable Identification 3 (henceforth, SV-Ident) shared task aims at promoting the developing of systems that can identify variables within the text of scholarly publications from the social sciences in different languages (initially, we focus here on English and German). The shared task is divided into two sub-tasks: a) Variable Detection and b) Variable Disambiguation. The former deals with identifying sentences that contain variable mentions, while the latter focuses on linking the correct variables mentioned in a sentence. Variable mentions are often implicit (e.g., sentences 1 and 3 in Figure 1), and understanding when a variable is mentioned may require contextual information as well as knowledge from external sources (e.g., a variable vocabulary). Since annotating scientific texts requires domain knowledge, training data is costly to create and thus scarce. To overcome these limitations, NLP systems, e.g., models using pre-trained language models (PLMs) and transfer learning are promising technologies to use.
In this paper, we report the results on the first edition of the SV-Ident shared task. Two teams made a total of 9 submissions to the leaderboard. One of the teams developed systems for both sub-tasks and submitted a system description paper. While none of the teams improve on the baselines, we use the submissions provided by the teams to collect a few initial findings on the difficulties and challenges of the SV-Ident task. Crucially, we find that there is a difference between the performance on two types of variable mentions: explicit and implicit. Implicitly mentioned variables (sentence 1 in Figure 1) are significantly more difficult to detect and disambiguate, as they require contextual knowledge. This opens up new research questions for future work, such as, for instance: can implicit mentions of survey variables be further categorized into finergrained classes or can co-reference resolution be used to link variable mentions across different parts of a document? In order to foster future research on this task, we release all of our code to reproduce the analysis results and the annotation guidelines for creating the dataset at https://github.com/ vadis-project/sv-ident.
The remainder of this paper is organized as follows: we provide an overview of the dataset used in §2. In §3, we describe the task definition and evaluation metrics. We present the submitted systems in §4 and provide a detailed analysis of the results in §5. We briefly discuss related work in §6 and frame the shared task into a broader context in §7. Finally, we summarize the shared task and propose future work in §8.

Data
The SV-Ident 2022 shared task has been conceived in the context of the VADIS project 4 and organized as part of the third Workshop on Scholarly Document Processing (SDP) (Chandrasekaran et al., 2020), co-located at the 2022 International Conference on Computational Linguistics. In the following, we describe the data collection process and the dataset used for the shared task.

SV-Ident Corpus
The SV-Ident Corpus contains publicly-available scientific publications from the Social Science Open Access Repository (SSOAR) 5 in full text. To collect the corpus, we first filter the 5,000 most popular research datasets using search logs from GESIS Search. 6 We then retrieve the publications linked to these datasets as our candidate set. Finally, only those publications that had at least one associated research dataset with indexed variables on the GESIS Search platform are retained. This results in 285 documents from the original set of 120k publications. For this set of candidates, we then select 44 documents for annotation, which include the most popular ones as well as publications linked to variable vocabularies of different sizes.
Each document in our dataset has been annotated in PDF format using the INCEpTION software (Klie et al., 2018) by two domain experts (graduate students trained in the social sciences). Annotators are provided with the whole document and asked to label all sentences that contained variables, including the variables the sentences mentioned. We first conduct two calibration rounds, for which annotators are given 50 two-page documents from the dataset collected by Zielinski and Mutschke (2018). Afterwards, the selected 44 documents are annotated in three annotation rounds over a period of 8 weeks (on average, each annotator spent between 1-2 hours on each document). Texts are then extracted, and all parsing errors are manually corrected. words. Because annotators have access to all parts of the document at once, the annotation setup allows the use of document-level knowledge to infer sentence-level labels. The annotations include the variable IDs that are mentioned in a text from a set of possible candidates, confidence scores for the annotations, and, for the test set, annotators also classified each mentioned variable into an explicit or an implicit mention (examples of explicit and implicit mentions were both found in the annotation guidelines). We generally define explicit mentions as those which do not require contextual information to be labeled correctly. The opposite is true for implicit mentions.

SV-Ident Shared Task Dataset
When annotating, the set of candidate variables is potentially made up of all variables from the research datasets linked to a publication on the GESIS platform: this set usually contains hundreds or thousands of variables, thus making the annotation task impractical and hard to scale. To help reduce the size of the set of possible survey variable labels, annotators are provided with a tool to find matches using different methods. The first method uses an ensemble of four sentencetransformer models to predict the top 20 variables that are semantically most similar to the reference sentence for each model. The annotators receive recommendations for variables for which at least two models predict them to be in the top 20 results. The second method allows annotators to search using a method of matching strings approximately rather than exactly: specifically, we use the Token Set Ratio metric, which compares the number of insertion and deletion operations for unique and com-  mon words between the strings to be compared. 7 The last method simply provides annotators with the full list of variables to manually search through. All three methods have their drawbacks. The first two might fail to recommend valid variables for cases with high linguistic variation, vagueness, or infrequent words, while the last may provide annotators with a search space that is too large. While we do not control for such possible failures, future work may draw insights from the analysis of the annotations.
The dataset for the shared task is a subset of the SV-Ident corpus. More specifically, 14 out of the 44 annotated documents from the SV-Ident Corpus are additionally filtered out due to missing links to research data, incorrect annotations, or PDF parsing errors, leaving 30 documents in total. The dataset consists of 18 documents (7 English and 11 German) for the training and development sets and 12 documents (6 for each English and German) for the blind test set.
An example of a sentence and its metadata, including annotated labels from the dataset, is shown in Figure 2. Each instance in our dataset contains: a document ID (doc_id); a binary label (is_variable), where a value of 1 implies that the sentence contains a variable; the language of the sentence (lang); a list of document-level linked research datasets (re-search_data); the sentence (sentence); a unique ID (uuid); and a list of annotated variables (variable). Raw sentence counts for each of the dataset splits are provided in Table 1. Since the test set contains more English sentences, during evaluation, we compute the mean of the scores for each language for the competing systems (see §5 for more details). In total, there are 3,823 training, 425 validation, and 1,724 test sentences. English and German sentences are roughly evenly distributed at 3,035 and 2,937 instances for each language, respectively.   Because of the challenging nature of the annotation task, we join the annotated instances of variable mentions and link variables of each annotator. We provide agreement scores between annotators for English and German instances separately (Table 2). We calculate the Cohen's Kappa score for agreement on Variable Detection. The scores for both English and German range between 0.46 and 0.48, which indicate that there is a moderate agreement. For Variable Disambiguation, we use Krippendorff's Alpha. Both languages have an agreement score of 0.08, which implies that the agreement is close to random. One reason for such low agreement is the large number of possible variables to choose from, given that the total vocabulary size for all the documents is very large (27,365 variables that are often similar). The annotators labeled 1,165 unique variables (around 4% of the vocabulary) a total of 11,356 times (Table 3). In the future, we plan to analyze this disagreement with respect to the choice of variables further. Looking at the document-level, variables occur with different frequency in different documents (shown in Table 4). The size of the variable vocabulary (i.e., the subset of all variables, containing only the variables from the research datasets that are linked to a publication) related to a publication ranges from 64 to 5,733. The number of annotated variables is at least 13 and at most 1,204 for English, and for German 20 and 1,143, respectively. The number of uniquely annotated variables is at most 153. In the final analysis, we investigate at which ratio sentences of a document are annotated. While the annotation ratio is at least 7%, it is at  most 86% for relatively dense documents. 8 In addition to document-level differences, variables may require contextual knowledge to be disambiguated. Based only on the test set, annotators agree that 242 sentences had explicit, 13 implicit, and 18 both types of mentions. At the fine-grained annotator-level, the first annotator labeled close to 37% more implicit than explicit mentions, while the second labeled nearly thirteen times as many explicit as implicit mentions. Given that we did not conduct calibration rounds on this specific concept, annotators may not have shared the same understanding, since this distinction was introduced only in the third round of annotations. Future work will focus on further analyzing and validating the annotations. We make our dataset available on GitHub as well as on HuggingFace. 9 In addition, we also release, as the trial dataset, the data that were originally created by Zielinski and Mutschke (2018) (while the annotation procedure does not follow the same guideline, the data can be used as additional training data). Notably, consecutive sentences mentioning the same variable as well as vague variable mentions were not annotated in the trial data. We manually filter the trial data, after which, 446 English and 573 German sentences remain in the training set and 87 and 111 in the test set for each language, respectively.

Experimental Setup
The task of SV-Ident deals with identifying variable mentions in a text. For simplicity, the task is formulated as a sentence-level task, but can also be solved using document-level information (inline with the data annotation process). The shared task is decomposed into two sub-tasks: Variable Detection and Variable Disambiguation, where the former task can be used to help filter candidate sentences for the latter.

Tasks
Task 1: Variable Detection. The first task can be seen as a binary text classification task. More formally, given a set of texts T (in our case, sentences), for each t, where t ∈ T , systems should predict the binary label l ∈ [0, 1] for t, where a value of 1 implies that t mentions a variable.
Task 2: Variable Disambiguation. The second task can be viewed as an information retrieval (IR) task, where the goal is to identify all relevant documents (i.e., variables) for a given query (i.e., input sentence). More formally, given a set of queries Q that mention variables, where Q ⊆ T , and the set of all documents D (in our case, variables), for each q, where q ∈ Q, systems should predict the subset of documents D that are mentioned in q, where D ⊆ D.

Evaluation Metrics
To evaluate systems, we use standard text classification and information retrieval evaluation metrics. For the first task, systems are evaluated using the standard F 1 − macro score averaged across languages and documents. F 1 − macro is defined as follows: where P and R are the precision and recall scores, respectively. The F 1 − macro averages the scores for P and R across classes (i.e., scores are computed for each class separately and each is weighted equally). For the second task, systems were evaluated using the (Mean) Average Precision (MAP) score with a recall cutoff value of 10 (denoted as MAP@10). Average Precision (AP) measures the average of the precision scores at each relevant item returned (i.e., recall level) in a search result set. MAP is the mean of the AP scores when computed across more than one query. MAP considers the ranking position of each relevant document. It further assumes that a user desires to retrieve many relevant documents. MAP is defined as follows: where P is the precision score, K the recall level, and N the number of queries. We choose MAP over accuracy, because MAP incorporates the rank of the predicted document, which accuracy ignores.
In a realistic use-case, a user may be interested in being recommended up to K relevant variables per sentence. While we did not empirically test what value of K would be most suitable for a user, we choose K to equal 10, since 95% of all sentences are labeled with up to 10 variables. In addition to F 1 − macro and MAP@10, we provide secondary metrics, which are not used for ranking the submitted systems, but can provide additional insights into the results. These include precision (P), recall (R), different values of K for MAP, and R-precision, which is the precision at recall R, where R is the number of relevant documents for a query.
In order to account for dataset imbalance during evaluation, for each score function f (i.e., evaluation metric), we compute the average score across languages and documents. The intuition is that languages and documents are equally important, and a model should perform well on all. The average score is computed as follows: where L is the set of languages, D the set of documents, and D l the set of documents for a given language, for l ∈ L and D l ⊆ D.

Shared Task Setup
The shared task was hosted on CodaLab. 10 After registering for the shared task, participants could download the test set and were asked to submit their predictions on CodaLab as a single file for each task (submissions were allowed from July 18th through August 1st, 2022). Submissions were limited to 20 for each task. For each submission, an automated evaluation system would upload the computed scores to the public leaderboard.
Two teams participated in our challenge on Co-daLab, and one of the teams submitted a system description, which is included in the proceedings. We summarize the report here. The participant (Hövelmeyer and Kartal, 2022) treated both tasks, at least partly, as a problem of semantic textual similarity (Agirre et al., 2013). For Task 1, sentences were first preprocessed by randomly undersampling in order to balance the data, removing stopwords, lemmatizing the data, and using only a subset of the fields from the vocabulary metadata based on preliminary experiments. Then, test sentences and vocabulary data were converted into dense sentence representations using Sentence-T5 (Ni et al., 2022) for English and Sahajtomar/German-semantic 11 (henceforth, GS) for German. Similarity scores were computed for those test sentence and vocabulary item pairs. Pairs with a score greater than a predetermined threshold were classified as sentences containing variables. For Task 2, the same sentence representations were used, but for all test sentences. The variables were then ranked based on their scores, with a higher score implying a greater similarity. While other methods were also implemented, such a Logistic Regression and Multinominal Naive Bayes classifiers, the best performing systems used Sentence-T5.

Evaluation
This section first describes the baseline systems for each task and later provides the results of the shared task.

Baselines
We train a transformer-based model for Variable Detection and implement lexical and neural zeroshot baselines for Variable Disambiguation.
The baseline system for the first task uses a transfer learning approach by fine-tuning a pre-trained language model (PLM) on the training and validation datasets. We use a PLM that was further pre-trained on a corpus of English social science abstracts, SsciBERT (Shen et al., 2022), which outperforms BERT (Devlin et al., 2019) and SciB-ERT (Beltagy et al., 2019) models on the SV-Ident test set. Because no multilingual or German PLM counterparts exist that have been pre-trained on scientific texts, we use the specialized monolingual SsciBERT for both English and German data.
For the second task, we implement three baseline systems in a zero-shot setting: a lexical as well as sparse and dense retrieval models. We choose BM25 as our lexical baseline, using Elasticsearch. 12 For the sparse model, we use SPARTA (Zhao et al., 2021) and a multilingual sentencetransformer 13 (Reimers andGurevych, 2019, 2020) as the dense retriever. Rather than training the models on the data, we use them to first encode the query and documents (i.e., variable metadata) and later rank those which are most semantically similar to a query by computing the cosine similarity between query-document pairs. The similarity computation assumes that instances that are closer together in vector space are semantically more similar. While participant 2 conducts an ablation study on the choice of metadata to use for matching the variables, we choose to include all metadata and leave finding the the optimal combination of metadata to future work.

Results
Task 1 had two participants and a single baseline system, while Task 2 had one participant and three baseline systems. In the tables below, the systems are denoted as follows: participant 1 as Unk, participant 2 as S-T5/GS (or S-T5 for English and GS for German), the baseline for Task 1 as SSBert * , and the baselines for Task 2 as BM25 * , Sparse * for the SPARTA model, and Dense * for the multilingual sentence-transformer (all baselines across text and tables are always marked with a * asterisk).
Variable Detection. For this task, none of the participating systems are able to beat the average score of the baseline. Unk scores lower than chance likelihood, while, with a score of 60.17, S-T5/GS comes close to SSBert * , which has a score of 66.10 (Table 5). Breaking the scores down into the average scores across documents for each language, S-T5 outperforms the baseline for English. Thus, Task 1 can also be solved in a zero-shot setting, given that the Sentence-T5 model was not finetuned on the provided data. Similar large PLMs may show further improvements.  At the document-level, systems show varying performance (see Table 8 in the Appendix). For the document with the ID 21357, participants' systems have low scores, while SSBert * has the highest score across all documents. Furthermore, for 7 out of the 12 documents, the baseline system has the highest score. In addition to the number of positive and negative instances, we also report the number of variables associated with a document as well as the year of the publication. When computing the Pearson correlation coefficient, we find a weak correlation between the F 1 scores and the size of the search space (i.e., vocabulary size) for Unk (r = 0.132, p = 0.68), S-T5/GS (r = 0.266, p = 0.26), and SSBert * (r = 0.152, p = 0.64). With respect to the year of the document, we find a moderate correlation for Unk (r = 0.272, p = 0.39), S-T5/GS (r = 0.274, p = 0.39), and SSBert * (r = 0.392, p = 0.21). However, these correlations may not generalize due to the small number of documents.
Given the low annotator agreement with respect to the fine-grained labels, explicit and implicit, we report scores for the cases where both annotators agree on the label (see Table 9 in the Appendix) as well as for each annotator independently (see Tables 10 and 11 in the Appendix). We divide the labels into explicit, implicit, and mixed classes, where sentences that contain explicit and implicit variables are labeled as mixed. In cases where both annotators agree on the label, systems perform better on explicit than on implicit or mixed mentions. The same is true for annotator 1, except for S-T5/GS. This implies that explicit mentions are easier to detect and disambiguate. This is not the case for annotator 2. A possible explanation could be the low number of implicit annotations, which may be due to a difference in understanding of the labels. Unk outperforms all systems for the cases when both annotators agree on the label. This is surprising given the low average performance of the system (unfortunately, no system description was provided).
Variable Disambiguation. For the second task, we report only a single submission together with the results for three baselines ( Table 6). As described in Section 5, the baselines include BM25, SPARTA (henceforth, Sparse * ), and a multilingual sentence-transformer (henceforth, Dense * ). While we provide participants all the test sentences, we only evaluate performance on the subset of instances that contain variable mentions, as Task 1 already validates Variable Detection performance (this setup ignores false positive queries submitted by the participants). Unless explicitly stated, the following discussion mainly focuses on the MAP@10 scores. While the participant's system performs close to Dense * for English, Dense * scores twice as high for German. Sparse * outperforms all systems on English data. This is likely due to the system having been trained on a large English retrieval corpus. 14 BM25 * and Sparse * perform worse on German. Lexical models, such as BM25 * , are prone to perform worse for languages that have many rare words, such as German, which allows compound nouns. Furthermore, because Sparse * is only specialized for English, it does not perform well for data in a different language. Overall, Dense * outperforms all systems by at least 0.5 points for English, except for Sparse * , and by at least around 10 points for German.
At the document-level, scores vary significantly (see Table 12 in the Appendix). Scores across different values of K improve as K increases. For dense documents (i.e., documents with a high ratio of variable mention sentences), scores increase significantly when going from k = 1 to k = 5, such as for the IDs 21357, 57204, and 66324. Furthermore, while some systems perform well on a document, others perform poorly. For example, the document with ID 66324 shows the lowest performance by all systems except for BM25 * , which has a score of 22.01 and is the second-highest document score for BM25 * . For 57561, BM25 * achieves only a score of 1.60, while all other systems score higher than 16. S-T5/GS outperforms all baselines only once and twice when compared  to only Dense * . Such exceptions may be caused by a larger overlap between the tokens in the document and the underlying data used to train the models. In addition, we find a moderate correlation between MAP@10 scores and the vocabulary size (and a strong correlation for Dense * ) for S-T5/GS (r = 0.395.p = 0.20), BM25 * (r = 0.465.p = 0.13), Sparse * (r = 0.427.p = 0.17), and Dense * (r = 0.623.p = 0.03). As the search space increases, performance goes down. Finally, we find that MAP@10 is highly correlated with R-Precision (r = 0.941, p = 4.99), which implies that MAP is a good metric in the absence of the ground truth number of relevant variables. Performance on the annotator-level is similar to that of Task 1: scores are highest when both annotators agree on the label (see Table 13 in the Appendix). For both annotators, scores for the explicit class are consistently higher than for either implicit or mixed classes (see Tables 14 and 15 in the Appendix). This means that for the task of Variable Detection, knowing whether a variable is mentioned explicitly or implicitly can mean a 10 to 20 point absolute difference in performance. In the case when either both annotators agree on the label or when looking only at annotator 1, Sparse * outperforms all systems. Exploring other sparse models is a promising future direction for disambiguating implicit variable mentions.

Related Work
Identifying mentions of survey variables in text was first introduced by Mutschke (2017, 2018)   As the predecessor of our task, they created the first dataset for the problem of SV-Ident. Table 7 shows the statistical differences between the OM and SV-Ident datasets. Although fewer documents are annotated in SV-Ident, the number of instances in SV-Ident is almost 5 times that of OM. To have a greater diversity of survey variables, SV-Ident corpus uses 76 datasets with more than 27k variables from different research studies, such as ALLBUS, ISSP, and Eurobarometer, whereas OM only used a single dataset. Moreover, the SV-Ident corpus comes up with modified and additional annotation features: the unknown (UNK) token was used for ambiguous variable mentions; consecutive mentions of the same variable were included; confidence levels of the annotations and variable mention types were labeled; and variables were linked across languages. As a result, our corpus is much larger and more diverse.
Given that identifying variables requires semantic relations, other NLP tasks deal with a fundamentally similar perspective, such as entity linking (EL), recognizing textual entailment (RTE), semantic textual similarity (STS), plagiarism detection, and detecting previously fact-checked news. EL can be conceptualized as linking mentions to variables in a knowledge base (Rao et al., 2013). Since there are many similar survey variables in research datasets, disambiguating the right variable for a sentence is similar to determining the identity of an entity from a knowledge base. The RTE task is to identify whether a sentence entails a given candidate hypothesis or not (Dzikovska et al., 2013). A question answering adaptation of RTE (Dagan et al., 2013) is similar to SV-Ident, as the question and each answer form a hypothesis, which then re-quires the system to determine whether a sentence entails a given candidate hypothesis. STS is yet another similar task, which aims to find the similarity level between given texts (Agirre et al., 2013). STS was organized as a shared International Workshop on Semantic Evaluation between 2012 and 2017, and STS models have been developed for various domains (Wang et al., 2020;Yang et al., 2020;Guo et al., 2020). In the task of Plagiarism Detection of PAN, 16 a system should extract all plagiarized passages from a given set of candidate documents with (external) or without (intrinsic) comparing them to potential source documents (Potthast et al., 2013). Lastly, Detecting Previously Fact-Checked Claims, a shared task by the CheckThat! Lab (Nakov et al., 2022), aims to match the most similar claimstext fragments from social media or political debate scripts -to a corpus of verified claims. The corpus is used to find the most similar claims, which does not require direct linking, as is done in SV-Ident Task 2, because implicit links are inferred.

Why SV-Ident?
Today's search engines are the core elements of information access for social scientists. While search engines have seen many improvements in terms of keyword search and text understanding, they suffer from a limited capability of retrieving information from interconnected data sources, such as academic literature and research datasets. Nonetheless, they show outstanding performance on retrieving such documents individually. Current interlinking infrastructures typically only link research datasets to publications on the citation-level. Such systems do not yet consider fine-grained linking of publications to individual survey variables from research datasets. As demonstrated in the SV-Ident shared task, survey variables may be mentioned implicitly, which makes their manual or automatic identification non-trivial. Currently, social scientists have to manually identify such variables, which is timeconsuming. In addition to these limitations, search engines do not yet support queries specific to social science topics, concepts, or relations. Yet, keyword search, which is widely used, has many known problems (e.g., vocabulary mismatch or complex queries). As a result, social scientists are unable to access interlinked publications and research data. Thus, the re-use and reproducibility of research is limited.
SV-Ident, and more generally the VADIS project, plays an important role in filling the gap in the lack of infrastructure for social scientists (Kartal et al., 2022). SV-Ident aims to build automatic models for identifying survey variables in social science publications. This directly enables a more finegrained interlinking of publications and research datasets. More specifically, variables can be linked on the sentence-level, which allows new features to be developed. Within the VADIS project, we aim to develop variable-based automatic summarization, which will allow scientists to quickly get an overview of a publication with respect to the variables used. Furthermore, we plan to incorporate variable recommendation algorithms into the GESIS Search platform to enable scientists to find relevant variables outside the scope of variables they are already familiar with.

Conclusion
This overview reports on the results of the SV-Ident 2022 shared task. We introduce two subtasks relevant for SV-Ident, namely, Variable Detection and Variable Disambiguation. We report on data, which is currently the largest of its kind, that was collected, annotated, and made publicly available for this challenge. Baseline as well as participants' systems are described and evaluated. We find that nearly all systems perform better on explicit variable mentions, opening up new directions of research. Finally, we contextualize the shared task into related work and highlight its importance within a broader context. Future work will further analyze the distinction between different variable mention types. In addition, multi-task learning could solve both tasks jointly or in combination with adjacent tasks. Co-reference resolution could be used to help disambiguate implicit variable mentions. Finally, evaluating systems on more diverse metrics, such as fairness or robustness, is critical for applied research.  In the subsequent section, we focus on two theoretical concepts -mere exposure and hostile media perceptions -both of which have not been employed within the context of European identity.
The dependent variable (identity) was operationalized by two indicators: Citizens were asked to answer the following question: "Please tell me how attached you feel to the European Union" by using a scale from 1 (not attached at all) to 4 (very attached).
Furthermore, the item "You feel you are a citizen of the EU" was used in index building.
The independent variable was operationalized by a combination of several items: The participants were first asked the following question: "Could you tell me to what extent you…a) watch television on a TV set or via the Internet, b) listen to the radio, c) read the written press d) use the Internet?" Later they were questioned: "Where do you get most of your news on European political matters? Firstly? And then?" Possible answers were respectively: Television, the Press, Radio, and the Internet.
The respondents were asked, "Do you think that the [national] television present(s) the EU too positively, objectively, or too negatively?" The same question was repeated regarding the radio and the press. Even the unintended, casual contact with news on the EU provided by the media fostered European identity.    Table 9: Fine-grained results across types of variable mentions for Task 1. Sys = system, P = precision, R = recall, # = number of (positive) sentences.  Table 10: Fine-grained results across types of variable mentions for annotator 1 for Task 1. Sys = system, P = precision, R = recall, # = number of (positive) sentences.