{'fraction_non_alphanumeric': 0.026256224535989137, 'fraction_numerical': 0.011147578089633318, 'mean_word_length': 4.902805611222445, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An End-to-End Food Image Analysis System

Modern deep learning techniques have enabled advances in image-based dietary assessment such as food recognition and food portion size estimation. Valuable information on the types of foods and the amount consumed are crucial for prevention of many chronic diseases. However, existing methods for automated image-based food analysis are neither end-to-end nor are capable of processing multiple tasks (e.g., recognition and portion estimation) together, making it difficult to apply to real life applications. In this paper, we propose an image-based food analysis framework that integrates food localization, classification and portion size estimation. Our proposed framework is end-to-end, i.e., the input can be an arbitrary food image containing multiple food items and our system can localize each single food item with its corresponding predicted food type and portion size. We also improve the single food portion estimation by consolidating localization results with a food energy distribution map obtained by conditional GAN to generate a four-channel RGB-Distribution image. Our end-to-end framework is evaluated on a real life food image dataset collected from a nutrition feeding study.


Introduction
Dietary assessment refers to the process of determining what someone eats and how much energy is consumed during the course of a day, which is essential for understanding the link between diet and health. Modern deep learning techniques have achieved great success in image-based dietary assessment for food localization and classification [1,2,3,4,5,6,7,8], as well as food portion size estimation [9,10,11,12,13,14,6]. However, none of these methods can achieve food localization, classification and portion size estimation in an end-to-end fashion, which makes it challenging to integrate into a complete system for fast and streamlined process.
Image based food localization and classification problems can be viewed as specialized tasks in computer vision. The goal of food localization is to locate each individual food region for a given image with a bounding box. Pixels within the bounding box are assumed to represent a single food, which is the input to the food classification task. Food localization serves as a preprocessing step since it is common for food images in real life to contain multiple food items.
However, accurate estimation of an object's portion size is a challenging task, particularly from a single-view food image as most 3D information has been lost when the eating scene is projected from 3D world coordinates onto 2D image coordinates. An object's portion size is defined as the numeric value that is directly related to the spatial quantity of the object in world co-ordinates. The goal of food portion size estimation is to derive the food energy from an input image since energy intake is an important indicator for diet assessment. There are existing methods [13,14] that can estimate food portion size for the entire input image by generating a food energy distribution map, however, they cannot estimate the portion size of each food item separately. This is important as an individual food item can vary greatly in the energy contribution leading to significant estimation error. In this work, we address this problem by using a four-channel RGB-Distribution image, where the individual energy distribution map is obtained by applying food localization results on the entire food energy distribution map generated using conditional GAN as described in Section .
The success of modern deep learning based methods also rely on the availability of training data. Currently, there is no available food image dataset that includes groundtruth bounding box information, food category and corresponding portion size value for each food item in the image. Groundtruth portion size information is difficult to obtain from crowd-based annotation on RGB images, unless these numeric values are recorded during image collection. To address this issue, we introduce an eating occasion dataset containing all the groundtruth information listed above and the food portion size is provided by registered dietitians. We will describe the collection of this dataset in Section .
The main contributions of this paper can be summarized as follows.
• We propose an end-to-end framework for image-based diet assessment that integrates food localization, classification and portion size estimation • We introduce a novel method for single food item portion size estimation by using a four-channel RGB-Distribution image, where the individual energy distribution map is obtained by applying food localization results on the entire food energy distribution map generated by conditional GAN • We introduce a new food eating occasion image dataset containing bounding box information, food category and portion size for evaluating the proposed end-to-end framework

Eating Occasion Image to Food Energy Dataset
Annotated image datasets have been instrumental for driving progress in many deep learning based applications such as food detection and classification. Existing food images datasets may contain groundtruth bounding box and food label information [16,17] or just the food label [18,19] which is not suitable Figure 1: Example of an eating occasion image in our dataset: each food item is manually cropped containing the groundtruth bounding box coordinates and food category. All food and beverages were pre-weighed. A fiducial marker is used to calibrate the color and size of the input image [15].
for portion size estimation due to the lack of groundtruth information. In this paper, we introduce an eating occasion image to food energy dataset containing bounding box information, food category and portion size value. Food images were collected from a nutrition study as part of an image-assisted 24-hour dietary recall (24HR) study [20] conducted by registered dietitians. The study participants were healthy volunteers aged between 18 and 70 years old. A mobile app was used to capture images of the eating scenes for 3 meals (breakfast, lunch and dinner) over a 24hour period. Foods are provided in buffet style where pre-weighed foods and beverages are served to the participants. Based on the known foods and their weight, food energy is calculated and used as groundtruth. The dataset contains 154 annotated eating occasion images, with a total of 915 individual food images which belong to 31 categories. The corresponding groundtruth information includes bounding box to locate individual food, food category and portion size (in Kcal). The bounding box is given by the coordinates of input image as [x 1 , y 1 , x 2 , y 2 ] as shown in Figure 1.

Data Augmentation
We split the dataset with 15% for validation 15% for testing and the remaining for training. The problem with a small dataset is that the models trained on them cannot generalize well for data from the validation and test set. Hence, these models suffer from the problem of overfitting. Data augmentation is an efficient way to address this problem, where we increase the amount of training data by rotation (90 degrees, 270 degrees) and flip (x-axis, y-axis, both). We randomly implemented the operations based on the number of training images for that category, i.e. we implemented less operations for the category which contains more images. We augment the training data while keeping the groundtruth information unchanged before and after the augmentation operations.

Food Localization and Classification
The goal of food localization is to locate individual food region for a given input image by providing a bounding box, where each bounding box should contain only one food item. Deep learning based methods for localization such as Faster R-CNN [21] have shown success in many computer vision applications. It proposes potential regions that may contain the object with bounding boxes. Advanced CNN architectures such as VGG [22] and ResNet [23] can be used as the backbone structure for these methods.
The localization network locates all individual food items within the input food image and then sends them to the classification network. We apply Convolutional Neural Networks (CNNs) to classify the food item within each bounding box, which has been widely used in image classification applications. We use cross-entropy loss L c for classification task as shown below: where x is the cropped food image andŷ is its corresponding one hot label for the food category, f c denotes the output of classification with dimension n. The food localization and classification pipeline are described in Figure 2.

Food Portion Size Estimation
Portion size is a property that strongly relates to the presence of an object in 3D space, so it is very difficult to accurately estimate an object's portion size by given an arbitrary 2D image. In [14], a synthetic intermediate result of 'energy distribution' image was proposed, where the 'energy distribution' image has pixel-to-pixel correspondence and weights at different pixel locations to represent how food energy is distributed in the eating occasion. For example, pixels corresponding to steak have much higher weights than pixels of apple. [24] then uses the generated distribution image to estimate food portion size by applying a regression network. On the other hand, [6] uses RGB food image only and apply feature adaptation to estimate food portion size. Our method combines the two methods and use a RGB-Distribution image to improve the estimate of the food portion size.
Generate energy distribution map: We first train an energy distribution map generator by using a Generative Adversarial Networks [25] under conditional settings [26]. We define: where G is the generator, D is the discriminator, L L1 (G) is the L1 reconstruction loss, and L cGAN (G, D) is the conditional GAN loss as defined in [26]: where x is the source domain (RGB image), y is the target domain (energy distribution map) and z is random noise. The energy distribution map is a single-channel image where higher pixel value indicates higher energy distribution. Apply food localization bounding box: After we generate the energy distribution map for the entire eating occasion food image, we apply the bounding box generated in Section to obtain the energy distribution map for individual food item.
Generate RGB-Distribution image: We then combine the cropped RGB single food image with its corresponding energy distribution map to generate a RGB-Distribution image, which has four channels: R, G, B, and distribution map. The RGB-Distribution image is sent to a regression network to estimate food portion size. L1-norm loss L r is used for portion size estimation: Figure 2: The overview of our proposed end-to-end framework that integrates food localization, classification and portion size estimation. Given an input eating occasion image, the localization network locates each individual food item by generating a bounding box around the food region. Meanwhile, an energy distribution map is generated using conditional GAN. Then we directly apply a generated bounding box on an energy distribution map to get a corresponding energy distribution map for each food item. The cropped RGB food image is sent to a classification network to predict the food category. It is also used to generate the four-channel RGB-Distribution image by pairing the cropped RGB image with an individual energy distribution map, which are sent to a regression network to estimate portion size value.
whereŷ is the groundtruth portion size value and f r denotes the output of regression network with dimension 1. The lower half of Figure 2 shows the pipeline for estimating portion size for each individual food item.

Experimental Results
In this section, we evaluate our proposed end-to-end framework using the dataset introduced in Section .
For the localization and classification tasks, mean Average Precision (mAP) is the most common performance metrics. We firstly define several related terminologies: The intersection of union (IoU) refers to the ratio of overlapped region between predicted bounding box and groundtruth bounding box over the union of the two bounding boxes. True Positive (TP), False Positive (FP), True Negative (TN) and False Negative (FN). For example, TP means the predicted bounding box is assigned with correct food label and the corresponding IoU socre is larger than a threshold. Based on these definitions, we can calculate precision (Equation 5) and recall (Equation 6).
Average Precision (AP) for each category is the average precision value for recall value over 0 to 1 for each food category, and mAP is the mean value of all APs of all categories.
Since we use L1-norm loss as shown in Equation 4 to train the regression network, we use the Mean Absolute Error (MAE) to evaluate portion size estimation, defined as where w i is the estimated portion size of the i-th image,w i is the groundtruth portion size of the i-th image and N is the number of testing images.

Implementation Detail
Our implementation is based on Pytorch [27]. ResNet-50 is used as the backbone of Faster R-CNN. For regression network, a standard 18-layer ResNet is applied. The ResNet implementation follows the setting suggested in [23].

Results for localization and classification
The mAP results for food localization and classification tasks on our proposed dataset under different thresholds are shown in Table 1. 0.5 is commonly used and practical IoU threshold and we achieve satisfactory but 0.75 is a challenging threshold as we set the threshold of IoU > 0.75. In addition, our dataset is challenging since the number of training data is insufficient although some data augmentation methods are implemented. We also calculate the the mAP by changing the IoU threshold from 0.5 to 0.95 with a step size of 0.05 as shown in last column.

Results for portion size estimation
Compare to state-of-the-art methods: We compare our result of food portion size estimation with two state-of-the-art food portion estimation methods: [24] and [6] that directly using food distribution map or single RGB image for regression respectively as described in Section . The input for our proposed method to estimate food portion size is a generated RGB-Distribution image of cropped RGB image and cropped energy distribution image using the localization network. As shown in Table 2, our method outperforms the other two methods for single food item portion size estimation with smallest MAE as our proposed method takes into consideration for both the RGB and energy distribution information.
Compare to human estimates: We also compare our results for food portion size estimation of the entire eating occasion (a) (b) (c) Figure 3: Sample results for our proposed end-to-end image analysis system. The input to our system is a RGB food or eating scene image and the output contains the bounding box for each single food item along with the predicted category and portion size in unit of Kcal. The value inside () shows the groundtruth portion size. (Best viewed in color)

Methods
Mean Absolute Error (MAE) Fang et al. [24] 109.94 Kcal He et al. [6] 107.55 Kcal Our Method 105.64 Kcal image that containing multiple single food items with 15 participants' estimates from the same study. During the data collecting time, the participants are required to estimate the portion size of the meal they just consumed in a structured interview while viewing the eating occasion images. We sum up all single food portion size estimated by our proposed method, [24] , [6] and human estimates respectively for each eating occasion image. We apply error percentage as metric in this part which is defined as where w i is the estimated portion size andŵ i is the groundtruth portion size.

Methods
Error Percentage Human Estimates 62.14% Fang et al. [24] 35.06% He et al. [6] 25.32% Our Method 11.22% As shown in Table 3, the error percentage (EP) of human estimates is 62.14%, which also indicates that predicting food portion size using only information from food images is really an challenging task for majority of people. Our method gives the best result on EP for 11.22%, which improves more than 50% in terms of EP compared with human estimates. Figure 4 shows the results for each eating occasion image in test set. Our predicted energy (red dots) is most closest to the groundtruth energy (black line).

Conclusion
In this paper, we propose an end-to-end image-based food analysis framework that integrates food localization, classification and portion size estimation. We introduce a novel method to estimate individual food portion size using RGB-Distribution image, where the individual energy distribution map is obtained by applying localization results on the entire energy distribution map generated by conditional GAN. Our framework is evaluated on a real life eating occasion food image dataset with groundtruth information of bounding box, food category and portion size. For localization and classification, we calculate the mAP under different thresholds and we show a satisfactory result. Our proposed method for food portion size estimation outperforms existing methods in terms of MAE as we consider both the RGB information and energy distribution information when estimating the portion size using a regression network. Our method also achieves the best improvement of error percentage from 62.14% to 11.22% when compared with human estimates for the entire eating occasion image, showing great potential for advancing the field of image-based dietary assessment.