{'fraction_non_alphanumeric': 0.03136962465821526, 'fraction_numerical': 0.016803380561769825, 'mean_word_length': 4.70504821327283, 'pattern_counts': {'":': 2, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Quantifying Orphaned Annotations in Hypothes.is

Web annotation has been receiving increased attention recently with the organization of the Open Annotation Collaboration and new tools for open annotation, such as Hypothes.is. We investigate the prevalence of orphaned annotations, where neither the live Web page nor an archived copy of the Web page contains the text that had previously been annotated in the Hypothes.is annotation system (containing 20,953 highlighted text annotations). We found that about 22% of highlighted text annotations can no longer be attached to their live Web pages. Unfortunately, only about 12% of these annotations can be reattached using the holdings of current public web archives, leaving the remaining 88% of these annotations orphaned. For those annotations that are still attached, 53% are in danger of becoming orphans if the live Web page changes. This points to the need for archiving the target of an annotation at the time the annotation is created.


Introduction
Annotating web resources helps users share, discuss, and review information and exchange thoughts. Haslhofer et al. [12] define annotation as associating extra pieces of information with existing web resources while the Open Annotation Collaboration (OAC) group defines an annotation as a set of connected resources [12] where the basic form of annotation consists of the Body and Target resources. Ideally, the Body should be about the Target. Annotation types include commenting on a web resource, highlighting text, replying to others' annotations, specifying a segment of interest rather than referring to the whole resource, tagging, etc.
Hypothes.is 1 , an open annotation tool, was released in early 2013 and is publicly accessible for users to annotate, discuss, and share information. It provides different ways to annotate a web resource: highlighting text, adding notes, and commenting on and tagging a web page. In addition, it also allows users to share an individual annotation URI with each other as an independent web resource. The annotation is provided in JSON format and includes the annotation author, creation date, target URI, annotation text, permissions, tags, comments, etc.
One of the well-known issues of the Web is that Web pages are not fixed resources. A year after publication, about 11% of content shared on social media will be gone [15,16], and 20% of scholarly articles have some form of reference rot [13]. Lost or modified web pages may result in orphaned annotations, which can no longer be attached to their target web pages. Figure 1 shows the annotated web page http://caseyboyle.net/3860/ readings/against.html which has 144 annotations from Hypothes.is. The text with darker highlights indicates more users have selected this part of the page to annotate. The issue here is that all of these annotations are in danger of being orphaned because no copies of the target URI are available in the archives. Figure 2 shows the target URI http://climatefeedback.org/, created in December 2014, with the annotation "After reading about your project at MIT news, I visited your page and ..." on the highlighted text "Scientific feedback for Climate Change information online". In August 2015, this annotation can no longer be attached to the target web page because the highlighted text no longer appears on the page, as shown in Figure 3. Although the live Web version of http://climatefeedback.org/ has changed and the annotation was in danger of being orphaned, the original version that was annotated has been archived and is available at the Internet Archive (https://web.archive.org/ web/20141210121018/http://climatefeedback.org/). The annotation could be re-attached to this archived resource.
This paper is a follow-up to our paper [3] presented in August 2015 in the 19th International Conference on Theory and Practice of Digital Libraries (TPDL). In the TPDL version, we analyzed 6281 highlighted text annotations collected in January 2015, while in this paper, we worked with 20,953 annotations collected in August 2015. Figure 4 shows that the number of annotations in Hypothes.is has increased since July 2013. In this paper, we introduce a detailed analysis of the extent of orphaned highlighted text annotations in the Hypothes.is annotation system as of August 2015. We also look at the potential for web archives to be used to reattach these annotations. We find that 22% of the highlighted text annotations at Hypothes.is are not attached to the live web, and only a few can be reattached using web archives. Further, we show that 53% of the currently attached annotations could potentially become orphans if their live Web resources change, because there are no archived versions of the annotated resources available. Our analysis points to the potential for reducing orphaned annotations by archiving web resources at the time of annotation.

Related Work
Annotation has long been recognized as an important and fundamental aspect of hypertext systems [14] and an integral part of digital libraries [1], but broad adoption of general annotation for the Web has been slow. Annotations have been    studied for digital library performance [9,20] and methods have been explored for aligning annotations in modified documents [7], but typically such studies are limited to annotation systems specific for a particular digital library. While orphaned annotations of general web pages have been studied in the context of Walden's Paths [10,8], our study of Hypothes.is is a more recent evaluation of annotation and page synchronization in a widely deployed system.
Memento [22] is an HTTP protocol extension that aggregates information about the resources available in multiple Web archives. We can use Memento to obtain a list of archived versions of resources, or mementos, available in several web archives. In this paper, we use the following Memento terminology: -URI-R -the original resource as it used to appear on the live Web. A URI-R may have 0 or more mementos (URI-Ms) -URI-M -an archived snapshot of the URI-R at a specific date and time, which is called the Memento-Datetime, e.g., URI-M i = URI-R@t i -TimeMap -a resource that provides a list of mementos (URI-Ms) for a URI-R, ordered by their Memento-Datetimes There has been previous work in developing annotation systems to support collaborative work among users and in integrating the Open Annotation Data Model [18] with the Memento framework. The Open Annotation Collaboration (OAC) [12] has been introduced to make annotations reusable through different systems like Hypothes.is. Before publishing OAC, annotations would not be useful if the annotated web pages were lost because annotations were not assigned URIs independent from the web pages' URIs. By considering annotations as first-class web resources with unique URIs, annotation not only would become reusable if their targets disappear, but also would support interactivity between systems. Sanderson and Van de Sompel [19] built annotation systems that support making web annotations persistent over time. They focus on integrating features in the Open Annotation Data Model with the Memento framework to help reconstructing annotations for a given memento and retrieving mementos for a given annotation. They did not focus on the case of orphaned annotations and assumed that the archived resources were available in web archives. Ainsworth et al. have estimated how much of the web is archived [2]. The result indicated that 35-90% of publicly accessible URIs have at least one archived copy, although they did not consider annotations in their work, the result might estimate the number of orphaned annotations by factors like how frequently web pages are archived and the archiving process coverage. In other work [17,11] researchers built annotation systems that can deliver a better user experience for specialized users and scholars. The interfaces allow users to annotate multimedia web resources as well as medieval manuscripts in a collaborative way. In this paper, we focus on orphaned annotations and investigate how web archives could be used to reattach these annotations to the original text.

Methodology
We performed our analysis on the publicly accessible annotations available at Hypothes.is as of August 2015. The interface allows users to create different types of annotations: (1) making a note by highlighting text and then adding comments and tags about the selected text, (2) creating highlights only, (3) adding comments and tags without highlighting text, and (4) replying to an existing annotation. Table 1 shows how many annotations belong to each type.
In August 2015, we downloaded the JSON of all 33,946 publicly available annotations from Hypothes.is. Figure 5 shows the JSON of the annotation from Figure 2 with relevant fields shown in bold. The "updated" field gives the annotation creation date, "source" provides the annotation target URI, "type":"TextQuoteSelector" indicates that it is a highlighted text annotation,   "exact" contains the highlighted text, and "text" contains the annotation text itself. We focus only on annotations with highlighted text ("type":"TextQuoteSelector"), leaving 20,953 annotations for analysis. To determine how many of those annotations are orphaned, for each annotation we performed the following steps: -Determine the current HTTP status of the annotation target URIs ("source").
-Compare selected highlighted text ("exact") to the text of the current version of the URI. -Discover available mementos for the target URI.
-Search for highlighted text within the discovered mementos.
In Table 2, we show the top 10 hosts with annotations at Hypothes.is. Many of these hosts, including the top three, are academic servers and appear to use the system for annotation of scholarly work. Apart from this listing, we did not attempt to make judgements about the content of the annotations or annotation target text in our analysis.

Determining the HTTP Status
In the first step, the current HTTP status of annotation target URIs can be obtained by issuing HTTP HEAD requests for all URIs. In addition, we extended this to detect "soft" 401, 403, and 404 URIs, which return a 200 OK status but actually indicate that the page is not found or is located behind authentication [4]. One technique we used to detect "soft" 4xx is to modify the original URI by adding some random characters to the parent directory, so that it is likely that the new URI does not exist. After that, we download the content of the original URI and the new one. If the content of both web pages is 93% (or above) identical, and the HTTP status code of these URIs is "200 OK", then we consider that the HTTP status of the original URI is a "soft" 4xx. We have written a Python program available in GitHub 2 for detecting "soft" 4xx URIs.
The returned responses will determine the next action which should be made for every URI. The resulting responses can be categorized into three different groups. The first group contains URIs with hostnames localhost or URIs which are actually URNs. The second group has URIs with one of the following status codes: "soft" and actual 400, 401, 403, 404, 429 or Connection-Timeout. URIs with 200 status code belong to the third group.
The first group, localhost and URNs, were excluded completely from our analysis because these are pages that are not publicly accessible on the live Web. URIs in the second group, soft/actual 4xx and timed-out URIs, have been checked for mementos in the web archives. For URIs with response code 200, we have compared their associated highlighted annotation text with both the current version of the web page and the available mementos in the archives. Even though some annotations are still attached to their live web pages, we are interested to see if they have mementos to know how likely those annotations are to become orphans if their current web pages change or become unavailable.

Are Annotations Attached to the Live Web?
The second step is to compare the annotated text ("exact") of each annotation target URI that has a 200 HTTP status code with the current version of its web page and see if they match; this can be done by downloading the web page and extracting only the text which will be compared to the highlighted annotation text. Extracting text from a standard HTML resource is different from doing it with a Portable Document Format (PDF) file. We use curl to access and download web pages. Then, for HTML web resources, we extract only the text after cleaning it by removing all HTML tags, extra white-space characters, and others. If the web resource is a PDF file, extracting text will need one more step of converting the original binary contents to plain text. pdftotext 3 , which runs from the command-line and is freely available in many Linux distributions, can extract plain text from PDF files. After extracting the text either from a standard HTML web page or a PDF file, we search for the highlighted annotation text. If the text is not found, the annotation is considered not attached. For example, as shown in Figure 3, the annotation text is no longer attached to the web page as the highlighted text, shown in Figure 2, has been removed from the live web page.
For all annotations that are not attached until this point, we use PhantomJS to download the web page again trying to capture more resources that are loaded by JavaScript [5,6].
It is important to mention that when examining the existence of the annotated text in the related web pages, there are false negatives. In other words, our program might not identify some truly attached annotations because of reasons including but not limited to, not detecting "soft" 4xx URIs, failure in extracting text from PDF files by pdftotext, missing embedded resources in web pages that are loaded by JavaScript even with the use of PhantomJS. For example, the annotation that is about the highlighted equation in Figure 6 can not be re-attached to the the PDF file 4 because the highlighted equation is saved with a different encoding. Figure 7 illustrates another example where a web page 5 needs authentication to access the missing content. We can consider this example as an undetected "soft" 4xx as the server in this case should respond with 401 HTTP status code, but it returns "200 OK" instead.

Discovering Mementos for All Valid URIs
The third step is to discover mementos for all valid annotation target URIs. For this purpose, we used a Memento Aggregator [21,22] which provides a TimeGate through which we can get mementos that are closest to a URI-R's date. It would Fig. 7: Example of a Web Page (http://proquest.Safaribooksonline.Com/ book/programming/scala/9781449368814/preface/ who this book is for html) That Needs Authentication be a time-consuming task to check all available mementos for a URI-R to see whether they can be used to recover web pages. For example, URIs like http: //www.nytimes.com/ or http://www.cnn.com/ have thousands of mementos in different archives. The strategy that we use here is efficient in terms of execution time. For each URI, we only retrieve the nearest mementos to the annotation's creation date ("updated"). More precisely, we are capturing the closest memento to the date before the annotation was created and the closest memento to the date after the annotation was created.
In Figure 8(a), the annotation A was created at the time t 5 . The closest memento to the date before t 5 was M 2 (captured at t 4 ) while the closest memento to the date after t 5 was M 3 (captured at t 7 ). So, for this annotation we picked the two closest mementos, which are M 2 and M 3 . Figure 8(b) is an example where mementos are only available before the annotation creation date while in Figure  8(c), mementos are only available after the annotation creation date. It is also possible that an annotation target has no mementos at all, as Figure 8(d) shows. If there are multiple closest mementos from different archives that share the same creation date (memento-datetime), then we consider all of these mementos for two different reasons. First, it is possible that at the time a memento is requested from an archive, there would be a technical problem or server-related issue which may affect returning the requested mementos. Second, we would like to know how different archives could contribute to provide mementos and recover annotation target text.  The final step is to see whether annotated URIs can be recovered by their mementos. The same technique introduced in Section 3.2 is used to test mementos. If the annotation target text ("exact") matches the text in the discovered memento, then we consider that this annotation is attached to the memento. Otherwise, we consider that the annotation cannot be attached.

Results
We collected 33,946 annotations from Hypothes.is. Table 3   annotations. We noticed that a group of 820 annotations should be excluded from our analysis. This group consists of annotations whose target URIs are unresolvable, such as localhosts and URNs. In our further analysis, we will focus only on the 20,133 annotations that contain highlighted text and have resolvable target URIs. We noticed also that out of 20,133 annotations, 10% (1966) have URI-Rs that are no longer available on the live web, returning 400 and 500 status codes. Figure 9 shows how annotations are classified based on the status codes of their target URIs. After checking each annotation, we found that 15,773 (78%) of the highlighted text annotations are still attached to their live web pages as Figure 10 shows. This means that the remaining 22% of the annotations are orphans if there are no mementos that can be used to reattach these annotations.
Next for each annotation, we checked the archives for the presence of mementos of the target URI near the annotation creation date. In Table 4 we consider annotations that have mementos both before ("L") and after ("R") the annotation date. "No" under the L and R columns means that annotation cannot be attached to the nearest memento, while "Yes" means that the annotation attaches to the nearest memento. Table 5 shows the number of annotations that have mementos only on the L side (before) of the annotation date, and Table 6 shows the number of annotations that have mementos only on the R side (after) of the annotation date. Finally, Table 7 illustrates the number of annotations whose targets have no mementos. From these tables, we see that 4360 (22%) of the annotations can no longer be attached to their live web pages. Unfortunately, the current holdings of web archives only allow 12% of these to be re-attached, while the remaining   Table 7, the majority of annotations have no mementos available at all. Those that can no longer be attached to their live web version are lost (orphans), but those that are still attached are in danger of being orphaned (41% of all annotations). These annotations can be recovered if these pages are archived before the annotated text changes. As we can see, 60% of annotations  are either orphaned or in danger of being orphaned. Figure 11 shows the status of current Hypothes.is annotations. Table 8 shows the number of annotations that can be recovered using various archives, split by whether or not they are still attached to the live web. As expected the Internet Archive can be used to recover the most annotations.

Conclusions
In this paper, we analyzed the attachment of highlighted text annotations in Hypothes.is. We studied the prevalence of orphaned annotations, and found that 19% (3813) of the highlighted text annotations are orphans while 41% (8357) are in danger of being orphaned. We used Memento to look for archived versions of the annotated pages and found that 3% (547) of annotations that are not attached to the live web can be reattached to archived versions. We also found that for the majority of the annotations (11,273), no memento exists in the archives. This points to the need for archiving web pages at the time of annotation.