{'fraction_non_alphanumeric': 0.03524852040014665, 'fraction_numerical': 0.013827057036610276, 'mean_word_length': 4.730492196878751, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Complex and Precise Movie and Book Annotations in French Language for Aspect Based Sentiment Analysis

Aspect Based Sentiment Analysis (ABSA) aims at collecting detailed opinion information according to products and their features, via the recognition of targets of the opinions in text. Though some annotated data have been produced in challenges as SemEval, 
resources are still scarce, especially for languages other than English. We are interested in enhancing today’s mostly statistical text classification with the use of linguistics tools, in order to better define and analyze what has been written. The work presented in this paper focuses on two French datasets of movies and books online reviews. In reviews, text length is much higher compared to a tweet, giving us the opportunity to work on a challenging and linguistically interesting dataset. Moreover, movies and books are products that make classifying opinions into aspects quite complex. This article provides an analysis of the particularities of the two domains during the process of collecting and annotating data, a precise annotation scheme for each domain, examples and statistics issued from the annotation phase, and some perspectives on our future work.


Introduction
The Web makes it relatively easy to collect a large amount of textual data. Researchers, as well as enterprises, politicians and governments are very interested in analyzing them, especially to learn more about people's opinions. This is due to many reasons: to improve services and products, to know the public opinion, to avoid terrorism acts. For researchers, opinion analysis is a multifaceted problem, with different names according to the studied aspect: subjectivity analysis (Wiebe et al., 2004), opinion mining (Pang and Lee, 2008), sentiment extraction (Das and Chen, 2007). While the subject is mature, as proved by the many published surveys (Pang and Lee, 2008;Liu, 2012), there is still room for improvement, demonstrated by the interest for the yearly NLP conferences and workshops as SemEval or Wassa (Mohammad and Bravo-Marquez, 2017) and by the challenges that opinion analysis still offers (Breck and Cardie, 2016;Mohammad, 2016). Aspect Based Sentiment Analysis (ABSA) aims at "determining the orientation of sentiment expressed on each aspect" (Liu, 2012). ABSA was introduced as a shared task for the first time in SemEval-2014 (Pontiki et al., 2014), with datasets in English language for two domains: laptops and restaurants. In SemEval-2015, the task was repeated and extended, adding the hotel domain with a dataset of whole reviews and not just isolated sentences (Pontiki et al., 2015). In SemEval-2016 (Pontiki et al., 2016), new and multilingual datasets were provided: restaurant reviews in six languages (English, French, Dutch, Russian, Spanish and Turkish), hotel reviews in Arabic, consumer electronics reviews in three languages (English, Dutch and Chinese), telecom reviews in Turkish and museum reviews in French. In this paper we propose an annotated French dataset for different domains compared with SemEval ones: movies and books. We have chosen to create a corpus of reviews and not tweets because we are interested in combining statistical and linguistical approaches, therefore we need to work on texts longer than tweets. As far as we know, there are few works in French for these domains (Hamdan et al., 2016), as a consequence it is not so easy to find this kind of data. Using ABSA is a difficult task on such reviews, because opinion is expressed in complex and various forms. Unlike other kind of reviews, limited in total amount of usable characters, these reviews are non-predictable in terms of length, and may carry opinions about other products related to the reviewed one, which are used as comparison. They can also merge in a same paragraph user's opinion and description of the evaluated product. The paper describes our corpus (see Section 2.) and the specific form of annotation we have chosen in order to cope with comparisons and language complexity (Section 3.). Finally, we briefly explain how our future work will exploit these annotated data (Section 4.).

Collecting text reviews
In our first experiments, the corpus was only composed by a collection of 450 books reviews and 450 movies reviews from the French Sentiment Corpus (FSC) produced between 2009-2013 by Vincent and Winterstein (2013). According to that paper, FSC was not originally used for Aspect-Based Sentiment Analysis, but to study opinion polarities. We decided to extend the corpus by adding more recent reviews -the NC Corpus, created between 2016 and 2017. The final corpus is the result of the union of these two corpora, both with equal dimensions. In total it is composed by 1800 reviews, i.e 4113 sentences for the books domain and 5222 for the movies domain. The corpus comes from two websites: Amazon.fr for books, and Allocine.fr for movies. By taking into account word and line counts of similar projects, it is possible to state that the corpus used in this project is suitable for being used in the context of Aspect-Based Sentiment Analysis tasks. Compared to other corpora in the field, the one used in this project has a higher-than-average number of sentences. Some examples are (Hamdan et al., 2016) with 200 books reviews in French, (Alvarez-Lopez et al., 2017) with 2977 sentences from English books reviews, (Thet et al., 2010) with 1000 sentences from English movies reviews, (Sorgente et al., 2014) with 2648 sentences from movies reviews in Italian. The corpus deals with books and movies using three types of rating: 1 for extremely negative reviews, 3 for the moderate ones, and 5 for the extremely positive ones. We decided to include 3 stars rating reviews because we noticed that, in such reviews, reviewers do not express a strong opinion: they are therefore induced to justify their balanced opinion to precise what they consider to be the negative and positive aspects of the book or movie. The corpus provides the title of the reviewed product, to detect if the movie or book analyzed is the same of the title, or if it is another one used as term of comparison. While making NC corpus, we noticed that collecting negative books reviews was not so easy, because there was an overwhelmingly higher amount of positive reviews compared with the negative ones. We suppose that, unlike people reviewing movies, people who don't like the book that they are reading prefer to close it and not to write a review -unless they have read others author's works and they want to criticize the author's choices compared to the previous ones. Concerning the linguistics point of view we noticed that, more than in movies reviews, the book genre constitutes a linguistics sub-domain of books, i.e. user's style changes based on the genre of the book that is reviewed. For this reason we chose, when possible, to collect reviews of different types of books, in order to have a more complete annotation. In effect we noticed two phenomena: books' reviewers seem to make fewer misspellings than movies' reviewers; in addition to this, it seems that there is a difference between the diversity of words used in movies and books reviews, even when they share the same topic. The difference is to be found in the fact that, in movies reviews, users do an evaluation by just labeling the movies as good or bad (following a description of the several aspects of the product). In books reviews, instead, users try also to develop a critique that uses a variety of context-specific words. In Table 1  ) and a general vocabulary to evaluate a movie and not specially designed for the topic (un beau film, j'ai aimé les images magnifiques, les décors somptueux (...) y avait la musique vraimant belle [a good film, I loved the wonderful images, the gorgeous sets (...) the music was really good]). On the other hand, in books reviews sometimes it is harder to find misspellings. Moreover, it is not just an evaluation about the strengths and weaknesses of the product (L'habilité de l'écrivain nous permet de voir la société dans laquelle nous vivons [the writer's talent lets us perceive the society in which we live]). The reviewer, in fact, develops a critique about the topic of the book using more context-related ex-

Corpus statistics
Each part of the corpora (divided per rating) contains 150 reviews.  Statistical data (see Table 3) show that the number of words of the reviews is very variable (some reviews have more than 2000 words). The great σ indicates a great dispersion of the values and the difference between mean and median is the clue of an asymetrical distribution.  The histogram showing the number of words of movies and books reviews (see Figure 1) indicate that most of the reviews are composed by less than 200 words. A box-plot reveals that, for the movies domain, reviews with more than 254 words are to be considered outliers, and the average amount of words per review is 127.5. On the other hand, for the books domain, reviews with more than 181 words are to be considered outliers, and the average amount of words per review is 96.5. Another difference between the two parts of the corpus is that, even if movies reviews are generally longer than books reviews, they are less dense. In other words, we noticed that book reviewers are more creative in language expression than movie reviewers, as it is possible to see from the table 4 that shows the ratio between the total number of words (token) and the unique words (types) on the different parts of the corpus divided per type and rating. Movies' users tend to write longer reviews when they are giving a three-star review to a product (M3 has 43289 tokens, compared with negative reviews -31595 tokens, and positive -22658 tokens).
FSC corpus, despite the domain, shows a number of tokens lower than the new one (exception made by the FSC B3 reviews). We suppose it is due to the increasing comfort for the user in being critical compared with the past. In fact, it is common to find very long reviews reposted from blog articles.

Annotation
The objective of the annotation is to provide a both generic and precise tool, which makes it possible to: 1. study (or automatically learn) the vocabulary used to express a negative or positive opinion, generally or in a given aspect; 2. know indications of the aspect related to a given positive or negative sentiment.
At the same time, we wanted the annotation to offer a generic aspect classification, which can be applied to various kinds of books or movies.

Annotation scheme and guidelines
We tried to create annotation schemes suitable for all type of movies or books, regardless of genre. We did not provide an annotation scheme specially designed for e-books. This is due to two reasons: first, we did not find many big differences between a book review and an e-book review; second our corpus does not have e-books reviews. The annotation scheme is composed by aspects and attributes (see Table 5 and 6). The annotation scheme for the book domain is composed by 5 aspects and 19 attributes. The movie one is composed by 7 aspects and 28 attributes. These classifications may be viewed as very precise, but grouping classes is easy, depending on the expected use of the corpus.

Aspects Attributes General Feeling Text
General, Subject, Style, Characters, Pace/Narration, Interest/Accuracy, Translation/Adaptation, Readability Illustration General, Interest/Accuracy, Graphic quality Author General, Text Author, Translator, Illustration Author, Form General, Bookbinding, Typography, Inner structure, Distribution

Opinion annotation
Each opinion expression is annotated in three steps.
1. The first step is to select a group of contiguous words that indicate a positive or negative opinion. Opinion is evaluated by an ordinal value: -1 or -2 for a negative sentiment, according to its intensity; 1 or 2 if the sentiment is positive.
2. The second step is to detect the entity to which the opinion has to be reported. This entity is not always expressed, especially if it is the movie or book that is evaluated. When it is expressed, it is most of the time a name or a nominal group. Since including coreference resolution is beyond the subject of this work, pronouns are not selected as entities. Whenever opinion expression refers to a pronoun, the entity is reported of its previous closest reference. If the entity is detected, a relation is created, which joins opinion expression with entity phrase.
3. In the third step, an aspect and an attribute are chosen in the annotation scheme.
The many forms of opinion expressivity are related to various relations between entity, opinion word and aspect, as pointed out by the following examples.
• In the very simple phrase "Un bon film" [a good movie], "bon" [good] indicates a positive sentiment (value: 1), the entity is film [movie] and the aspect General Feeling is chosen because of the entity.
• In "c'est un navet" [it's a rubbishy movie] the word "navet" [rubbishy movie] indicates a very negative sentiment (value: -2) and refers to the entity at the same time. So, the aspect, given by the entity, is General Feeling.
• In the phrase "Le style est très agréable" [the style is very pleasant], extracted from the book corpus, the phrase "very pleasant" indicates a very positive sentiment (value: 2). The entity is "style". It indicates that the category is Text with Style attribute.
• In the same corpus, the phrase "le livre est bien maĺ ecrit" [the book is very badly written], the phrase related to the sentiment is "bien malécrit" [very badly written] (value: -2), the entity is "livre" [book]. The aspect is Text with Style attribute, because of the verb ecrire [to write].
• In a very negative book review, "la bobo au style frelaté" [the boo-boo with degenerated style], the word degenerated refers to a very negative opinion (-2). It can be reported to the entity Style and classified in Text#Style). Because of the reference to the style, one can say that bobo refers to the author; like in "un navet", "la bobo" expresses in a single word the entity and the opinion of the reviewer.
Previous examples, though being very simple, show how entities, opinion phrases and context have to be combined, to determine the aspect to which they have to be reported.

Entities related to other products
Some phrases indicate a positive or negative sentiment related to another book or movie, most of the time to be compared with the reviewed one. For example, in "rienà voir avec le seigneur des anneaux, carrément passionnant" [(this film has) nothing to do with the Lord of the rings, (that is) downright fascinating], (1) the phrase "downright fascinating" indicates a very positive opinion, but it is applied to another movie: "the Lord of the rings". On the contrary, the full phrase indicates a negative feeling about the movie. Such comparisons are frequent and can be a problem for an automatic opinion detection; that is why we wanted the possibility for the annotation to report them precisely. To cope with the problem, the annotation of the entities indicates whether they are or not related to the evaluated product, a product of the same series, another product, etc. So, in the previous example (1), the very positive phrase "downright fascinating" is reported to "the Lord of the rings" classified as an entity which refers to another product. The phrase "(this film has) nothing to do with the Lord of the rings" is annotated as a negative opinion, reported to an entity which refers to the evaluated product.

Annotation Process
The annotation has been done by two experts: a native speaker and a non-native speaker. After the choice of the annotation form and the redaction of the guidelines, experiments have been conducted to estimate inter-annotator agreement. The most difficult task was the selection of the phrases related to an opinion, with particular attention to the determination of their scope. As per word selection, Cohen's κ was equal to 0.71, an acceptable result given the difficulty and subjectivity of the task. However, to improve the reliability of the corpus, we decided to perform a crossreading of the annotations between the two annotators.
The annotation was performed via Glozz software (Widlocher and Mathet, 2012). Glozz is a multi-purpose text annotation tool, which comes with a fully WYSIWYG interface. It makes it possible to create units, defined as contiguous span of texts and relations between them. Annotations may be exported in several file formats and especially as SQL data.

Annotation Results
We annotated 5001 opinion phrases on movies (M1, M3, M5) and 3274 on books (B1, B3, B5). Annotations on negative reviews outnumber annotations on positive reviews, with circa 1992 annotations on M5 and B5 corpora against around 2899 on M1 and B1 corpora.  In the book corpus, the most important six classes related to specific aspects are Text#Interest and Accuracy, Text#Pace-Narration, Text#Style, Text#Characters, Text#Readability and Text#Subject. All of them are related to the aspect Text and they collect 44.7% of the annotations not classified as General Feeling. In the very wide variety of the assessed books, the textual aspect represents therefore a very large majority, with great importance given to the interest in the content.
In the movie corpus, the most important five classes related to specific aspects are Acting#General, Script#Plot, Direc-tion#General, Script#Pace/Narration and Script#General. They collect 47.6% of the annotations not classified as General Feeling. Apart from Distribution, all the aspects collect a significant number of annotations: the cinema is a multi-modal media, which combines sound and images to tell a story played by actors. 6392 entities have been selected; 5693 of them are related to the book or movie on which the review was written; this means that around 5% of the opinion expressions are related to another book or movie: a little more than 3.8% for the movie corpus and slightly less than 6.9% for the book corpus. Since most of them are very different from the general opinion of the reviewed book or movie, it is interesting to detect them.

Perspectives and conclusion
The goal of the annotation is to give to the research community a dataset composed of annotated French opinion expressions about movies and books. The dataset has been created using the Aspect-Based Sentiment Analysis method, which is, based on our experience, the best fitted method to analyze these domains. Our future work will use our annotations on a system merging linguistics analysis and statistics. We will use a shallow parser and an opinion lexicon that takes advantage of the dataset to track complex expressions.
Datasets for aspect-based sentiment analysis in french. In Nicoletta Calzolari (Conference Chair), et al., editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may. European Language Resources Association (ELRA).