{'fraction_non_alphanumeric': 0.03529327199539965, 'fraction_numerical': 0.014447958596894767, 'mean_word_length': 4.655691056910569, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Concept Based vs. Pseudo Relevance Feedback Performance Evaluation for Information Retrieval System

This article evaluates the performance of two techniques for query reformulation in a system for information retrieval, namely, the concept based and the pseudo relevance feedback reformulation. The experiments performed on a corpus of Arabic text have allowed us to compare the contribution of these two reformulation techniques in improving the performance of an information retrieval system for Arabic texts.


Introduction
In Information Retrieval System (IRS), researchers work distinguishes two approaches for the Query Reformulation (QR). The first approach is called direct reformulation and consists of adding new terms to the initial query, it is based on external resources such as ontologies and thesaurus or the relation of cooccurrences between terms, i.e., Concept Based (CB) query reformulation [Baeza-Yates and Berthier, 1999, Carpineto and Romano, 2012, Efthimiadis, 1996. Whereas, the second approach, is called indirect reformulation and consists of modifying the user query by taking into account a list of documents already considered as selected and all the terms added to the query come only from the documents' collection and not from an external resource [Baeza-Yates and Berthier, 1999, Carpineto and Romano, 2012, Salton and Buckley, 1990]. This process is called Relevance Feedback (RF) once supervised and Pseudo Relevance Feedback (PRF) once automatic.
According to [Abderrahim and A, 2010, Harb et al., 2011, Xu et al., 2002, Baziz, 2005, Nathalie et al., 2008, Wedyan et al., 2012, the CB QR has a positive effect in information retrieval. In this same context, the studies applied in [Abderrahim and A, 2012, Xu and Croft, 2000, Kanaan et al., 2005, Salton and Buckley, 1990, Abderrahim et al., 2013 show that the RF (or PRF) is simpler to be realized and allows an improvement of IRS performances. This paper aims to evaluate and compare the contribution of these two reformulation techniques (CB and PRF) in improving IRS performance for Arabic texts.

Concept Based (CB) and Pseudo Relevance Feedback (PRF) query reformulation
Reformulating a query is an attempt to improve user queries by: adding terms; re-weight existing terms; removing terms; or a combination of those methods. In this section, we examine two approaches for improving the initial query formulation through query expansion. These approaches are automatic (i.e., user intervention is not required at any stage other than the formulation of the original query) and grouped in two main categories: (a) approaches based on information derived from knowledge structures such as thesauri (also called CB direct reformulation); and (b) approaches based on information derived from the set of documents initially retrieved (also called PRF indirect reformulation). In these approaches, an original query is run using conventional information retrieval techniques. Then, related terms are extracted from the top "n" (generally n=10) documents (also called local document set) that are returned in response to the original query. The related terms are then added to the original query, and the expanded query is run again to return a new set of documents, which are returned to the user. In the literature, two techniques are proposed for the extraction of terms from de the local document set [Baeza-Yates and Berthier, 1999]: (a) The local clustering: this consists of constructing a matrix of association that quantifies the correlation relations of terms got from the set of documents that were returned in response to the initial query.
According to the method of construction of the correlation relations, we notice three types of clusters: association clusters, metric clusters and scalar clusters.
(b) The local context analysis: it consists of using the concepts instead of the keywords to represent the document [Xu and Croft, 2000].
As mentioned above, PRF strategies are based on expanding the query with terms correlated to the query terms. Such correlated terms are those present in local clusters build from the local document set. The idea of local clustering approach is to build global structures such as association matrices which quantify term correlation and to use correlated terms for query expansion.
When using knowledge structures, expansion terms are determined from pre-fabricated term dependency matrices, and no significant work has to be done during query time to expand queries. During query time, queries are expanded simply by looking up related terms in the appropriate knowledge structures.
The CB QR (see figure 1) consists of analyzing the query aiming at detecting the terms which correspond to concepts in the external resource (Arabic WordNet (AWN) in our case). These terms are replaced by the proximate concepts by using the AWN semantic relations (only the synonymy relation is tested in our work).
The basic procedure of the CB QR is: -The user issues a query.
-The analyzer extracts terms which possibly correspond to concepts from the initial query.
-For each term extracted, the system try to selects all it proximate concepts from AWN (in our case, proximate concepts of a term "t" are synonyms of t from AWN).
-The query is then expanded with all proximate concepts.
-The system displays a set of retrieval results.

Figure 1 The CB query reformulation
The idea behind the CB QR is that for each term "t" in a query, the query can be automatically expanded with synonyms of "t" from AWN.
The PRF is a technique that consists of modifying the initial query of the user by adding some terms got from the list of documents retrieved in the IRS. The basic procedure of the PRF is: -The user issues a query.
-The system returns an initial set of retrieval results.
-The system selects the "n" first documents as relevant.
-The system computes a better representation (T terms) of the information need based on the "D" documents (also called local document set).
-The query is then expanded with all extracted terms.
-The system displays a revised set of retrieval results.
The PRF is based on three steps (see figure 2): -The samples: it consists of selecting a set of «D» documents (samples) among the returned ones by the IRS and judged as pertinent.
-The extraction of evidences: it consists of establishing the list of «T» terms judged pertinent for the expansion of the query.
-The rewriting of the query: It consists of enriching the query with terms found in the previous step.
According to the analysis of D and T for Arabic text in [Abderrahim et al., 2013], we have used D=15 and T=7.
The idea behind the PRF QR is that the documents retrieved for a given query "q" are examined at query time to determine terms for QE.

Reformulated
Query Analyzer

Select Proximate Concepts
Query Rewriting Information Retrieval System Returned Documents Figure 2 The PRF query reformulation In the following description of experimentation and discussion of obtained results are displayed.

Experiments' description
An external resource for research similar concepts in the phase of QR is used in our experiment. This resource is AWN, it is one of the most important lexical database of the Arabic language following the development process of Princeton English WordNet (PWN) [Fellbaum, 1998] and EuroWordNet [Vossen, 2004].
Like EuroWordNet, AWN is developed in two phases by: -Building a core wordnet around the most important concepts (Base Concepts). This core is highly compatible with wordnets in other languages that are developed according to the same approach.
-Extending the core wordnet to more specific concepts.
The AWN system falls into two parts: the AWN lexical database; and the software tools used to access the database. Two different web-based interfaces have been developed for AWN: A lexicographer's web interface and a user's web interface. The first one is designed to support the task of adding, modifying, moving or deleting AWN synsets. The second one enables the user to consult AWN and search for Arabic words, Arabic roots, Arabic synsets, English words, synsets offsets for English PWN.
The AWN Browser is a stand-alone Java application, it include browsing AWN, searching for concepts, and updating AWN with data from the lexicographers. Searching can be done using either English or Arabic. In Arabic, the search can be carried out using either Arabic script or Buckwalter transliteration and can be for a word or root form, with the possibly use of diacritics. The browser allows a user to navigate via hyponym and hypernym relations between synsets of AWN and PWN. It should be noted that the SUMO ontology navigation is also integrated into the browser. Users will be able to search or browse AWN using SUMO as

Initial Query
Pertinent Terms

Reformulated Query
Sampling (1) Extraction of evidences (2) Rewriting of the query (3) Information A corpus is also used in our experiment. It is composed of more than 22,000 documents containing Arabic texts from different fields. The main characteristics of this corpus can be found in Table 1.

Number of text files 22 824
Fields Health, sports, politics, sciences, religion, astronomy, nutrition, law, tales, family

Number of words 17 000 000
Number of terms 193 736 Three strategies are planned for information retrieval: -Simple Retrieval or without reformulation (SR): A set of 50 queries (see table 2) is used.
-Concept Based Retrieval (CBR): A set of 50 queries deduced from the initial queries by a blind enrichment is used. The set of the synonyms found in AWN is added to the initial query (see table   2). The following algorithm is implemented for this strategy.
-PRF Retrieval (PRFR): A set of 50 queries deduced from the initial queries by PRF enrichment is used (see table 2). The following algorithm is implemented for this strategy. It implements the local clustering technique to extract the most pertinent «T» terms that serve in the reformulation of the initial query.

Begin
For each query q i (i= 1,50)

Extraction of evidences
For each term t j of the query q i Find the list S j of t j synonyms from AWN

End
The retrieval results obtained by these various types of queries are saved in various files and various values of recalls and precisions of the system are calculated for each type of retrieval and query. Figure 3 presents the precision obtained at 5, 10, 20 and 100 documents (P@5, P@10, P@20, P@100) from different types of reformulation, and shows that the use of AWN did not improve the precision at different levels of documents after the QR. Moreover, the reformulation made by PRFR is much better than CBR. Table 3 presents the precisions at 11 levels of recall for each retrieval type.  Table 3 The precisions at 11 levels of recall for each retrieval type Figure 4 shows the curves' recall / precision obtained from table 4.

Figure 4 Comparison between recall / precision curves for each retrieval type
From figure 4 we can deduce that: -The PRFR has led to an overall improvement in the recall interval [0, 0.8].
-CBR has not led to an overall improvement in the performance of the Arabic IRS.
-The PRFR is more beneficial than CBR.
To understand the effect of different types of retrieval on each query, various measures that are mainly based on the comparison of results before and after expansion have been established.
For a given query, three cases can arise and are as follows: -Improvement (+): All precisions (at 11 points of recall) before are lower than those after. In other words, the curve (recall / precision) after is over before.
-No improvement (-): is the inverse of the previous case. The curve (recall / precision) before is over after.
-No decision (X): for some precisions, there is an improvement but for others there is no improvement. In other words, there is an intersection of the two curves (recall / precision).  Table 4 The indicator: Improvement (+), No improvement (-) or No decision (X), of each retrieval type.
From table 4 we can deduce that, whatever the retrieval type, we note that there is: -Improvement (+) in only two queries (queries number: 12 and 27) (4%).
-A set of 35 queries (70%) for which there is at least an improvement in 12 queries (24%). For the remaining 23 queries (46%), there is no improvement or indecision.
From the point of improvement view, the previous facts suggest that: -The reformulation by the use of an external resource or the PRFR can improve the performance of an Arabic IRS about 4%.
-There are queries that are hardly improvable with the QR.
In order to determine the best retrieval type (CBR or PRFR) the number of queries for each retrieval type has been counted (see table 5).
Results in table 5 can show that the reformulation by PRFR has the best rate of improvement (18%).
Furthermore, the analysis of the recall / precision curves (see figure 4) confirms this result.
Finally, we can conclude that the contribution of the use of an external resource in an Arabic IRS is about 14%. Moreover, it appears that PRFR is better than CBR.

Conclusion
In this article, CB and PRF, two techniques for QR in Arabic IRS are examined and compared. The results obtained, firstly, have confirmed that there are queries that are hardly improvable with the query reformulation and allowed measure the contribution (4%) of these two techniques in improving the overall performance of the Arabic IRS.
As for the queries' number that has really led to an improvement, comparison of results has showed that the PRF QR is better than CB QR. Moreover, in terms of recall / precision, the PRF QR is more beneficial than CB QR.
As for perspectives it can be said that this study has paved the way to test and compare other methods of reformulation with the same data of this experiment in order to determine the most appropriate technique to be adopted for Arabic IRS. [Abderrahim and A, 2010