{'fraction_non_alphanumeric': 0.02454071839868385, 'fraction_numerical': 0.005072662462297779, 'mean_word_length': 4.916058394160584, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A framework for food traceability information extraction based on a video surveillance system

Food security is currently one of the most concerning problems in China. A traceability system is an effective method to improve the quality of food production. This system has been widely applied in several countries such as the US, Japan and the EU. However, in China, the creditability of traceability systems is not strong and many producers try to deceive the public by forging the data in such systems. In this paper, a video surveillance system-based traceability system is proposed which will significantly increase the forgery cost. In this system, subjects, such as vehicles or people, are firstly defined using a novel dynamic background model, then their trajectories are generated and connected using different cameras with a camera relation graph. The experimental results indicate that the proposed method can efficiently extract the object information from the video surveillance system and generate image-based traceability information to be used for further analysis.


Introduction
Currently, the problem of food quality has gained increasing attention, especially in China. From rice to meat, from milk to wine, food quality problems are occurring one after another. One of the key technologies to improve food safety is to track the food production chain or, in other words, a food traceability system have been created. Nowadays, in countries such as the EU, USA and Japan, they have created food tracking systems.
For example, the UK has a system to record the status (the birth, growth and sale) of every cow via a Radio Frequency Identification (RFID) ear tag. In fact, RFID is the main method used to implement a traceability system and it has been used widely.
In spite of its many advantages, RFID data for a traceability system are difficult for the public to understand and trust. The reliability of RFID information has always been a problem. Since most traceability systems are built by their manufacturers, the data are easy to modify and forge by the manufacturing or operation company. Therefore, one needs a video surveillance system to confirm the traceability system to gain public trust. However, existing video surveillance systems are mainly designed for humans to watch, which is too costly for food production companies if they want to monitor the whole product process in real time. In fact, most video surveillance data are lost without any analysis.
In this paper, a framework to automatically extract traceability information from the video surveillance system is proposed. This framework first identifies the target object of interest and then finds the spatialtemporal trajectories of the target objects by the photogrammetry method. Finally, multiple trajectories from different cameras are joined together to form the whole traceability track of the target objects. In this traceability track, every moment is recorded by the camera which provides evidence for the public to check. Meanwhile, the video data are significantly compressed into the key images in the track, so long-term video traceability can be preserved for a low cost further use.
The proposed framework has the following advantages compared with the existing system: multi-trajectories from different cameras are combined as a whole traceable track. This framework is implemented in a grain company to track its rice production. According to the experiment results, the framework can efficiently and accurately extract the traceability trajectories of the people and vehicles. The rest of paper is structured as follows: the related work is listed in Section 2; Section 3 describes the proposed framework in detail; Section 4 describes the case study and the experimental results; Section 5 summarizes the paper and suggests future studies.

Traceability System
Recently, one of the most important research trends in the food sector has been electronic traceability and condition monitoring using RFID and Wireless Sensor Network (WSN) [1]. In practical implementations RFID has been applied in companies for food supply chains in Italy, France, the UK, Sweden, the USA and Canada [2~7]. Some electronic chain traceability systems have been proposed, such as the one in Frederiksen et al. [8] that proposed an Internet-based traceability system for fresh fish. Seino et al. [9] proposed a similar system for fish traceability by using QR codes after discarding the use of RFID due to the current cost of this technology. Grabacki et al. [10] introduced the concept of using RFID for the seafood industry in Alaska and they predicted that this would be the key technology in the supply chains of the future. More recently, research has demonstrated the use of RFID applications in the live fish supply chain [11], in intercontinental fresh fish logistic chains [12] and for monitoring the temperature of fish during the cold chain using RFID loggers [13]. The benefits of using RFID in the fish supply chain were also recognized by the Scandinavian fishing industry, with the main objective of developing and evaluating a traceability system [14]. With regard to WSN systems, Lin et al. [15] proposed a WSN-based traceability system for aquaculture that can automate many monitoring tasks and improve information flow.

Video Analysis
As previously discussed, it is not enough to only use RFID for a traceability system. Video surveillance should also be used to improve the credibility of the system. Two main types of semantic-based video retrieval for visual surveillance are studied: motion-based methods and semantic description of object motions.
Motion-Based Method: an online video retrieval system was proposed by Chang et al. [16] to support object-based indexing for spatiotemporal queries with algorithms for video-object segmentation and tracking. Principal component analysis (PCA) was also used to reduce the dimensionality of trajectory data [17], in which target objects were identified by a two-level PCA operation with coarse-to-fine retrieval. An object trajectory-based system was proposed for video indexing based on the Haar wavelet transform coefficients [18]. Wavelet decomposition was applied to find out each trajectory in the video [19]. Polynomial curve fitting was also used of motion model creation and indexing [20].
Semantic Description of Motions: A fuzzy sets based method was proposed to interpret dynamic-object interactions in temporal image sequences using fuzzy logic and fuzzy measures [21]. In [22], human activities in video images were described with a hierarchy of concepts related to actions. Scenarios were used to represent human activities and were translated into text by filling into a template of natural language [23]. In [24], descriptions of videos were generated with state transition models. Also in [25], a semantic-based surveillance video retrieval system was proposed.

Methodology
In the proposed framework, video surveillance data are analyzed and the interesting objects such as people or products are extracted to obtain their trajectories. The multiple trajectories of a target object from different cameras are connected as a complete traceability track according to their spatial distributions. In this section, the framework is discussed in detail.

Object Identification
Dynamic Background Model. To identify the target object in a traceability system, such as people, vehicles or goods, one first needs to retrieve the background in the surveillance system. Currently, there are many methods by which to obtain the background information. In this paper, a novel method is proposed to create a time-related background model to overcome the influence of sunlight and weathers. In the traceability application, the video camera is set still and monitors a certain place over a long period of time. Therefore, one can build a model of the background based on the long-term video. This model is generated as follows: first, a video clip, for example, is taken for 1 day. Then, one collects the frames every second (24*60*60 frames per day). Next, these frames are divided into different classes, backgrounds and targets. In this paper, the background class is defined as a set of frames: F_background={f1,…fn}, in which n>N, the timestamp of fi fi.t<fj.t if i<j, and the difference between two frames |fi-fj|<T. This means that the background has a certain number of frames which are similar to each other. This assumption does not hold for all video surveillance systems, especially those that have many targets appearing on the screen. Fortunately, in the food process and traceability industry, the background takes a large proportion of the total frames (more than 80% in our implementation). After the classification, a background set is generated for a video clip, and one frame is selected to represent the background. This background set will be used to identify the target object in the video clip.
Target Extraction. Based on the generated background set, one can find the start and end timestamp (ts, te) for each background. Then, for all frames in (ts, te), the corresponding background is selected to extract the target from the object frame, based on their difference. To minimize the influence of light and weather, the background is processed with histogram equalization.
Based on the generated dynamic background model, the target extraction is straightforward. Since the background is dynamically generated from the video clip (e.g. one day), accuracy can be guaranteed, even if there are some changes in the background. For example, if a truck stops in the field of view for a long time, it will be treated as a part of the background in the algorithm. Although this method is after the event processing, it is suitable for a traceability system in which there is no need for real-time analysis. If some systems require online target detection, many existing methods could be applied, such as optical flow or the frame difference method.

Multiple Camera Integration
It is necessary to combine multiple cameras into a unified system which can continuously monitor a target object from start to end. This is quite important for the traceability system. However, the volume of data from any surveillance system is extremely large and can not be processed by the food manufacturers. Since most food companies do not have a large IT department, it is possible for them to analyse three or four camera streams continuously, but usually there are more than 50 cameras in a surveillance system. In this paper, a multi-scale-based method to extract the traceability information from the large volume of video data gathered by the system is described.
First, the proposed continuous detection algorithm is run on one or a few selected video streams which reflect the overall situation of the monitored area. Then, whenever an interesting object is identified, video data between the detected event start and end timestamps from the relevant cameras will be retrieved for analysis. A camera relation graph can be predefined based on the location and orientation of the cameras. If two cameras have an overlapping area, then they are connected in the graph. Based on this graph, one is able to find all the events recorded in the surveillance system by simply monitoring one camera if the camera relation graph is a connected graph.

Video Surveillance System
Eight network video recorders (NVRs) are deployed in three factories. Each NVR is connected to 20-30 cameras distributed in different places and can record videos for three months. one can access these recorded data through the Internet with the supplied application program interfaces or APIs from the NVR. Fig. 1 shows an example of the NVR with 20 cameras. Currently, this monitoring system has run for 6 months, but most of the surveillance videos have been erased without any analysis because of the large volume of the video data. Every month, about 100 TB of data is generated from these NVRs and it is too costly to transform and process all of the data. This is quite a common problem in the application of a video surveillance system.

Object Detection
In order to deal with the huge volume of data from the surveillance system, several key cameras were selected to extract the trajectories of the target object which can reduce the network and computation load dramatically. A script was written to automatically download the video from the NVRs and analyse the frames in a centralized server. In the next step, these object detection methods were deployed in the NVRs to save the network traffic.
In this framework, the object is detected, based on the proposed dynamic background model, as described in Section 3.2. The implementation is based on OpenCV. The object detection results are listed in Fig. 2.  Fig. 3 shows the generated dynamic background model. From the background frames in different hours, one can see that the proposed method is able to select a suitable frame as the background to minimize the interference factors, such as weather, light or some stationary objects.

Multiple Camera Integration
Fig .4 shows the multiple camera integration results. By applying the proposed multi-scale analysis method, the overall computation load is reduced by 90%. For this factory with three NVRs and over 60 cameras, one only needs to fully analyze one camera, as shown in Fig. 3. All other information can be extracted from the predefined relationships among cameras.

Conclusion
In this paper, a video-based traceability framework is proposed which is a necessary supplementary feature for the current RFID-based system. In this framework, a dynamic background model was first designed to deal with complex background changes and improve the accuracy of target extraction. Second, a multi-scale video surveillance system is suggested to reduce the video analysis workload. Furthermore, a camera relationship graph is defined to extract the traceability video information from the whole system while only one or a few cameras are fully monitored. The proposed framework was partially implemented in a real rice production factory, and the experimental results suggested the efficiency of this system. In the future, based on the detected traceability data, an abnormal analysis method using a deep learning algorithm will be developed since the surveillance systemjj can generate a large volume of data. Also, an online query system will be built to allow the public to have better access to these traceability images.