{'fraction_non_alphanumeric': 0.04288541895755443, 'fraction_numerical': 0.019573345062678688, 'mean_word_length': 4.586609336609337, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Evaluation of OCR on Egocentric Data

In this paper, we evaluate state-of-the-art OCR methods on Egocentric data. We annotate text in EPIC-KITCHENS images, and demonstrate that existing OCR methods struggle with rotated text, which is frequently observed on objects being handled. We introduce a simple rotate-and-merge procedure which can be applied to pre-trained OCR models that halves the normalized edit distance error. This suggests that future OCR attempts should incorporate rotation into model design and training procedures.


Introduction
Optical Character Recognition (OCR) has been deployed in a wide range of environments, from document digitization [15] to road sign recognition [4]. Egocentric data presents a number of additional challenges, such as occlusion, motion blur, orientation and text size, particularly when objects are being handled [2].
In this short paper, we provide an initial evaluation of OCR on EPIC-KITCHENS data. In particular, we show that existing pre-trained models struggle with rotated text, which is common in real-life data, but missing from widelyused OCR datasets for model training as demonstrated in Fig. 1. We show that significant improvements can be obtained by rotating input frames and merging OCR results from multiple orientations. Note that this is not intended to be a definitive OCR method -the simplicity highlights a significant oversight in current state-of-the-art OCR methods, training datasets and procedures, suggesting directions for future research.
In summary, our main contributions are: • We annotate a subset of the EPIC-KITCHENS dataset to provide a ground truth for testing OCR on egocentric data. • We perform an evaluation of pre-trained state-of-the-art OCR methods on this dataset. • We investigate the effect of rotation on OCR accuracy, and propose a simple rotate-and-merge procedure which halves the character-normalised edit distance error. In Sec. 2, we give a brief overview of OCR methods and datasets. In Sec. 3, we introduce the EPIC-KITCHENS Text Subset dataset. In Sec. 4 we introduce the rotate-andmerge procedure, and present results in Sec. 5.

Background
This section presents a brief overview of OCR methods and datasets. OCR methods: OCR methods typically consist of two stages [14]. First, a detector finds text regions, which are cropped. Second, a recognition model extracts text from the cropped text region, effectively 'reading' the text.
Text detectors tend to follow the standard object detection paradigms. Some use single shot detection, similar to SSD [12]. A number of them use region proposals similar to the RCNN family of object detectors [17]. An advantage of region proposals is that they can handle a larger range of text sizes, which is especially important for egocentric data. In this work we use PSENet [24], which additionally exploits multiple scales (making it well suited to small text) as the detector due to its strong results on a number of baselines [8,26]. Related Efforts: Some attempts have been made at making recognition robust to rotation, through encouraging robust features [1] or generating rotated proposals for objects [17] and text [7], but these were not evaluated on the extreme rotations and distortions found in egocentric datasets (e.g. CT80 [18] and SVTP [16]), and their base architectures are outperformed significantly by more recent approaches [3,11]. OCR datasets: OCR datasets tend to focus on nearhorizontal text [8]. They come from diverse sources such as street view images [16] and synthetic generation [5]. Some have distortions [26], and varying text sizes [18], but none have the amount of rotation and distortion found in naturally-collected egocentric data. Methods are frequently trained on combinations of these datasets to improve generalisation performance [14].

The EPIC-KITCHENS Text Subset
To evaluate state-of-the-art recognition models and asses their suitability for indoor egocentric data, we curate the EPIC-KITCHENS Text Subset dataset. It contains 490 text cropped images 1 obtained from EPIC-KITCHENS-100 videos [2]. The crops were selected based on manual frame selection followed by detection confidence thresholding (from multiple rotations), ensuring varied examples are selected for labelling. Each detected text block was labelled by 5 Amazon Mechanical Turk Workers. A consensus was obtained by majority voting on the entire text content. Cases of multiple candidates were resolved manually.
The dataset contains 120 distinct word instances, and features a long-tail. Only 30% of the dataset contains image crops oriented around the horizontal (±15 • ), showing that a significant proportion of text is not horizontal. 1 Note that whilst 490 images is not suitable for training, it is sufficient to evaluate pre-trained models and demonstrate the effect of rotation.

Proposed Method
An initial inspection of OCR outputs from EPIC-KITCHENS images suggests that many of the failed recognitions are on text which is not horizontal, and that the majority of text is not horizontal. To address this, we introduce a simple procedure which exploits pre-trained OCR models. The same input image is rotated, forming a multi-rotation stack of images. These are all then passed through an OCR model individually. Their predictions are then rotated back to horizontal, to be directly comparable, and combined to produce one final output. This is shown in Fig. 2. We now discuss the method in detail.
We first define a set of rotations every r • as R = {ir : i = 0...360/r}, and their inverses as R −1 . We perform inference using a pre-trained OCR model Φ, which takes in an input image and returns a text bounding box b along with the recognised text t, bounding box confidence score µ and text recognition confidence score ν for each block of text it finds. We denote this Φ : R i (x) → {(b, t, µ, ν) ij : j = 1..J i } where J i is the number of discovered text blocks for the image x after it rotated by R ∈ R.
We apply Φ to all rotated images (so it acts on rotated text), which gives the set of all text blocks from all rotated images. We then apply the inverse rotations to the bounding boxes, so they align with the original, un-rotated image. B = {R −1 Φ(R(x)) : ∀R ∈ R} is the set of all text blocks from all rotations, mapped back to the original image.
Finally, we apply the Non-Maximum Suppression algorithm [17] to B, which takes in a set of boxes and confidence scores and returns the final set of detections for the image. We calculate the confidence score c ij for each box b ij as c ij = (µ ij + ν ij )/2.
• Average edit distance (Avg. ED): Edit distance per sample, averaged over all samples.
• Normalised edit distance (Norm. ED): Edit distance per sample, divided by sample ground truth length, averaged over all samples.
Results: Tab. 1 shows text recognition results across all methods with no rotation applied. Tab. 2 shows results for the same methods, when we integrate our proposed rotation pipeline. Clearly, all methods perform significantly better as part of the rotation framework. In particular, the best performing method (with and without rotation), ABI-Net [3], improves its accuracy from 16.5% to 49.6%, and normalised edit distance improves from 0.67 to 0.31. Examples are shown in Fig. 3. Additionally, the ranking of methods is consistent with and without rotation, which suggests that there is not a method which handles rotation significantly better than the others. This highlights a signifi-Method Accuracy ↑ Avg. ED ↓ Norm. ED ↓ ABINet [3] 49.6 2.43 0.31 SAR [11] 47.3 2.56 0.32 SATRN [10] 45.3 2.62 0.34 RobustScanner [25] 42.0 2.6 0.33 SegOCR [14] 31.0 3.27 0.37 NRTR [20] 23.7 3.67 0.48 TPS [23] 14.7 3.89 0.55 CRNN [21] 11.6 4.22 0.6 Table 2. Recognition results obtained with the rotation pipeline. Figure 3. Examples of text recognition by ABINet [3] with and without the proposed rotation procedure.
cant weakness in all current OCR approaches, and shows that rotation should be accounted for in future works. As expected, the best performing models are more recent and exploit transformers, compared to legacy network designs in the less successful methods.

Conclusion
In this paper, we investigated the ability OCR methods have to recognise text in images from the egocentric EPIC-KITCHENS dataset. We introduced an effective rotate-and-merge procedure which was applied to pretrained state-of-the-art OCR methods, and demonstrated large improvements against all non-rotation-aware baselines. This highlights a significant oversight of current OCR approaches. We also release our annotated data for future research at: github.com/tobyperrett/epictext-annotations Avenues for future work include exploring rotation as part of model design or training, annotating more text to enable fine-tuning models, as well as combining information from multiple frames to improve recognition accuracy. The ability to robustly recognise text in egocentric footage will provide a number of benefits, such as the ability to incorporate text into action and object recognition models.