{'fraction_non_alphanumeric': 0.02313985573897468, 'fraction_numerical': 0.005857578544803214, 'mean_word_length': 5.188735769922109, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sensor Fusion and Computer Vision Integrated System for Primary Separation Vessel Interface Level Estimation

In oil sands industry, primary separation vessel (PSV) is a critical component to recover bitumen from oil sands slurry. Accurate interface level estimation between froth and middlings layers ensures economical and environmental benefits of bitumen recovery. Nuclear density profiler, differential pressure (DP) cell, and image processing based computer vision system are usually used to estimate the interface level. The computer vision system, which uses a camera to capture sight glass vision frames, is considered to be the most accurate. Although the accuracy of computer vision system is high in normal operational conditions, its qualities are influenced by abnormalities, such as sight glass vision blocking, stains, and level switching between sight glasses. A sensor fusion approach, which recursively updates fusion parameters according to accurate computer vision results whenever they are reliable, is proposed. The fused results can then be used to provide reliable interface level estimation under abnormal scenarios. The sensor fusion algorithm is further integrated with computer vision system to improve frothmiddlings interface level estimation accuracy and robustness. Industrial environment simulations and factory accepted test (FAT) demonstrate the advantages and effectiveness of the sensor fusion and computer vision integrated system, which is applied in the industry.


INTRODUCTION
Primary separation vessel (PSV) plays critical role for bitumen recovery in oil sands industry. It is used to separate bitumen from oil sands slurry through a waterbased gravity separation process. General structure of PSV is a large open separation vessel, and a typical crosssection view of PSV is shown in Fig. 1. Oil sands slurry is pumped to the PSV for extraction through inlet pipeline. Then the slurry is divided into three layers due to gravity: the froth layer, middlings layer, and tailings layer, from top to bottom (Shafi et al., 2020). The froth layer has the lowest density and contains majority of bitumen. The middlings layer sits underneath the froth layer and mainly consists of clay particles that are difficult to separate. The tailings layer sits at the bottom of the PSV and is mostly comprised of coarse tails.
The exact location of the froth-middlings interface is a critical control parameter. Determining the interface level between the froth and middlings layer is important to the optimization of bitumen recovery that affects economic success of oil sands extraction (Tu et al., 2005). If the interface level is too high, middlings will get pulled into the overflow launder and thus reduce bitumen quality. If the interface level is too low, bitumen will get pulled into the middlings zone, which can reduce bitumen recovery efficiency. Common measurement instruments of interface level are nuclear density profiler, differential pressure (DP) cell, integrated sensor, and image processing based computer vision system. The nuclear density profiler based interface level detection approach was proposed in (Li et al., 2011). The profiler uses low energy gamma sources to emit gamma rays and uses detectors to receive them in order to measure the density at different points throughout the froth and middlings layers. However, both solid deposits and large air bubbles can block the radiation source and lead to profiler measurement errors (Jackson and Knapp, 2003). DP cell measures pressure at different locations in the froth and middlings layers and then refers to calibration sheet to determine the interface level. Owing to gradually varying process dynamics, the calibrated results may not ensure high-accuracy over a longer time period. An integrated sensor combines density and pressure measurements to provide interface level estimation. This measurement assumes fixed density at top and bottom of the two layers, which may not be consistent with real situations, and will lead to estimation errors.
In recent years, cost of camera installation has dropped, and accuracy of digital imaging software has improved, making image processing technique increasingly popular in the oil sands industry. In (Vicente et al., 2019), the computer vision system that combines static and dynamic image processing techniques to estimate interface level was proposed. The computer vision system improved image processing based interface level estimation accuracy significantly. However, this approach highly relies on sight glasses vision qualities. When camera is not able to capture a clear movement of interface level, the computer vision system will freeze its estimate by holding on to the latest interface level reading until the vision becomes reliable again.
Overall, fusion of interface level measurements from different sensors is desired in order to compensate for disadvantages of individual sensor and improve the robustness and accuracy of PSV froth-middlings interface level estimation. In this paper, a sensor fusion strategy, which recursively calibrates its fusion parameters, is proposed and integrated with the computer vision system to provide continuous reliable interface level estimation when computer vision results are unreliable. In addition to the PSV system, the proposed sensor fusion approach can be applied to general industrial processes, where both accurate but occasionally unavailable measurement and less accurate sensor measurements are available. Under normal conditions, high-accuracy measurement can be used to adjust fusion parameters of rest sensors recursively. When the highaccuracy measurement become unavailable or unreliable, fusion of other measurements will take over to provide reliable estimates continuously.
The remainder of this paper is organized as follows. Section 2 introduces the image processing based computer vision system and discusses existing problems. Section 3 explains the sensor fusion approach and fusion parameter update strategy in detail. The sensor fusion and computer vision integrated system is introduced in section 4, with updated implementation architecture. This section also demonstrates real industrial environment simulations conducted at the computer process control (CPC) industrial research chair (IRC) lab at the University of Alberta. Finally, section 5 concludes this paper.

IMAGE PROCESSING BASED COMPUTER VISION SYSTEM AND PROBLEM STATEMENT
The image processing based computer vision system for froth-middlings interface level detection in PSV has been developed to increase the interface level estimation accu-racy (Vicente et al., 2019). This approach combines static and dynamic image processing techniques, and provides indications when the results are not reliable. The image processing procedure relies on visions of the PSV sight glasses. A sample sight glasses vision captured by the installed camera is shown in Fig. 2 with annotations.

Fig. 2. Sample sight glasses visions of PSV.
The installed camera is rotatable to ensure it is properly aligned to the sight glasses. Typically, three sight glasses are installed on the PSV, named as upper, middle, and lower sight glass, respectively. The sight glass regions, which are named as regions of interest (ROI), are manually or automatically selected and marked by green rectangles when computer vision system starts. Similarly, the outer red rectangle defines reliability reference region (RRR) during the system initialization step. By defining ROI and RRR, the computer vision system is able to calculate quality and reliability indices.
Dynamic image processing method identifies interface level by finding the highest accumulative frame difference within ROI among the past few frames. The location where this highest difference occurs is determined as interface level and is marked using red line as shown in Fig. 2. The value of this highest difference is scaled to between zero and one and is named as quality index (Vicente et al., 2019). Several abnormalities can affect vision qualities and lead to low quality index. Sticky oil and sticky sands can cause stains on the sight glasses and lower the visibility of interface level (Jampana et al., 2008). Besides, when interface level crosses sight glass transition regions (yellow ellipse regions), dynamic movement of interface level is difficult to capture and low quality index will be issued.
Reliability index is calculated based on pixels intensity changes within the region of RRR excluding ROI. Significant difference between initial and current pixels intensity within this region will lead to low reliability index indicating significant changes in environment. Occasionally, operators need to perform maintenance on PSV and will partially or completely block the camera visions. This environment change will lead to large difference in pixels intensity and result in low reliability index. Fig. 3 shows sample graphical user interface (GUI) for the image processing based computer vision system (Vicente et al., 2019). The red ellipse marks the quality and reliability status. Both indication indices range from 0 to 1, and larger value means higher reliability. Thresholds of quality and reliability indices are user-defined according to desired system sensitivity. If either indication index is lower than its threshold, the computer vision system will freeze the latest reliable reading as a constant and warnings are issued. The objective of this work is to provide reliable dynamic interface level estimation during low-quality and low-reliability scenarios of the computer vision system using sensor fusion approach.

SENSOR FUSION
In this paper, a sensor fusion approach, which can be implemented in the computer vision system is proposed for industrial application. Under normal conditions, reliable computer vision results are used to adaptively calibrate the parameters of other sensors using the Kalman filter algorithm. Once the quality index or reliability index falls below their respective thresholds, the latest updated fusion model provides interface level estimation continuously.
Consider the discrete-time linear system: where x t is state, y t is output, and F t and H t are linear process model and output model, respectively; w t and v t represent process and measurement noises, which are assumed to be zero-mean uncorrelated Gaussian white noises, with Q and R as their covariance matrices, respectively.

The Kalman Filter Algorithm
For linear state estimation as described in (1) and (2), assuming the initial state estimation, x 0 , and error covariance, P 0 , are known or appropriately chosen, the Kalman filter algorithm can be implemented to recursively estimate the state through prediction and correction steps as (Simon, 2006;Xie et al., 2012):

Prediction:
In the prediction step, the predicted state and estimation error covariance are calculated as: wherex t andx − t+1 are posterior and prior estimates of the state, and P t and P − t+1 represent corrected and prior state estimation error covariance matrices.

Correction:
In the correction step, the Kalman gain is computed as: Then the posterior state estimate is obtained as: where y t+1 represents actual measurements, andŷ t+1 denotes model predicted output value, which is calculated as:ŷ (7) Then the corrected state estimation error covariance is

Fusion Parameters Update Strategy
In this study, update of fusion parameters is according to the computer vision quality and reliability status. Only when both indices are higher than their respective thresholds, the computer vision result is considered reliable and is used as output to recursively update fusion parameters using the Kalman filter algorithm. In this case, the fusion parameters are considered as process states, and random walk model is used to describe the process. Consequently, x t = [θ p,t θ dp,t θ is,t θ b,t ] T , and F t = I, where θ p,t , θ dp,t , and θ is,t are fusion parameters corresponding to nuclear density profiler, DP cell, and integrated sensor measurements, respectively, and θ b,t represents the bias term.
To compute the predicted output, the output model is H t = [P t DP t IS t 1], where P t , DP t , and IS t represent interface level measurements from nuclear density profiler, DP cell, and integrated sensor, respectively. Using reliable computer vision results as output measurements, the fusion parameters are updated through (6). If either quality index or reliability index is lower than its threshold, instead of freezing the interface level estimate, the most updated fusion parameters and bias are kept to continuously provide dynamic interface level estimation using (7). The fusion parameters update strategy can be summarized as: where QI and RI are the quality index and reliability index, with Q th and R th representing their respective thresholds.

Sensor Fusion Performance Test
To test the reliability of the sensor fusion results, both one-step ahead prediction and 10-hour prediction are calculated using one-month industrial data with sampling interval of every 10 seconds. In the 10-hour prediction test, computer vision results are assumed to be unavailable for 10 hours and the latest calibrated fusion parameters are used to provide interface level estimates during this prediction period. All data have been normalized for proprietary reason. In both types of testing, according to measurement accuracy, initial state and estimation error covariance are chosen as: x with Q = diag 1 × 10 −6 1 × 10 −6 1 × 10 −6 1 × 10 −6 and R = 1.
The correlations between results directly from computer vision system under reliable conditions (as reference) and measurements from individual sensors as well as from sensor fusion are presented in  Fig. 4 shows sample sensor fusion results over a 10-hour prediction period. The computer vision results, which use a camera to capture the sight glass visions, are plot as references, but they are not utilized to update fusion parameters during the 10-hour test period. As marked using red ellipses in Fig. 4, around the sight glass transition regions, where interface switches between sight glasses, the computer vision system is not able to determine the interface level accurately and has to keep the latest reliable estimate as its reading. The sensor fusion approach, however, continuously provides dynamic interface level estimation. Usually, interface level switching between sight glasses can take around half an hour, and regular PSV maintenance operation will not exceed a few hours, so the 10-hour estimation tests are sufficient.
From both correlation values and 10-hour prediction graphical illustration, it can be concluded that even without reliable computer vision calibration for 10 hours, the sensor fusion results still have higher interface level estimation accuracy than any other individual sensors. In addition, the sensor fusion results do not suffer from problems caused by low-quality sight glasses visions. Overall, it is reliable and effective to use sensor fusion results to take over froth-middlings interface level estimation under low-quality or low-reliability camera vision conditions.

SENSOR FUSION AND COMPUTER VISION INTEGRATED SYSTEM
In section 3, the sensor fusion results are proved to be robust and reliable under abnormal scenarios. In this section, the sensor fusion is integrated with the computer vision system, and the integrated system is implemented in actual process.
An improved computer vision system display for an industrial application is shown in Fig. 5. In the display, two boxes showing interface level readings are provided. The background color of the computer vision box changes from green to yellow under the low-quality conditions and changes from green to red under low-reliability conditions. The red line is the computer vision result, and the blue line indicates the sensor fusion result. In the computer vision system developed in (Vicente et al., 2019), a camera is used to capture and transmit visions of the sight glasses to the application computer, meanwhile a video processing application runs in the background to infer the level from video frames. On the basis of computer vision system, measurements from nuclear density profiler, DP cell, and integrated sensor are used to perform fusion algorithm through open protocol communication (OPC). In industrial implementation environments, the existing DCS may constraint complexity of the algorithms. To simplify this sensor fusion algorithm and reduce unexpected error resulted from data synchronization problems, only DP cell measurements are actually imported through OPC. The computer vision results when they are reliable are used to recursively calibrate the DP cell parameters and bias using (6) with x t = [θ dp,t θ b,t ] T . Under abnormal scenarios, we use (7) to provide dynamic interface level estimation using output model H t = [DP t 1] and the latest calibrated fusion parameters. The updated implementation architecture of sensor fusion and computer vision integrated system is shown in Fig. 6.
The factory acceptance test (FAT) was performed in the CPC IRC lab at the University of Alberta using actual PSV operating videos. The lab experimental set up is illustrated in Fig. 7. The monitor is used to display real sight glass vision videos. A digital camera is utilized to capture video frames, which are sent to the application computer, where the integrated system calculates both original computer vision results and sensor fusion results of the interface level. Fig. 8 shows a two-hour simulation with normalized industrial data. The first subplot shows the original computer vision results, DP cell measurements, and sensor fusion results. The second and third subplots indicate the corresponding quality and reliability indices from computer vision system. The red dashed ellipses mark the low-quality condition, which is mainly due to interface's crossing sight glasses. Over this period, the quality index falls below the threshold for around half an hour, and the computer vision freezes itself by holding on to the last reading of the interface level. Instead of holding the last reading as the computer vision system does, the sensor fusion algorithm uses the latest calibrated fusion parameters as well as DP cell measurements to provide interface level estimation continuously. The corresponding video images are illustrated in Fig. 9. In Fig. 9, the computer vision result is stuck at the boundary of middle sight glass, and background color of estimation value becomes yellow, indicating low-quality condition. During this period, the sensor fusion takes over to provide dynamic interface level estimation.
In addition, the blue dashed ellipses in Fig.8 indicate the low-reliability condition, which is caused by maintenance activities. The corresponding scenarios are shown in Fig.  10. In this figure, the operator was conducting routine check and maintenance on the PSV, and the sight glasses vision is completely blocked. In this case, the camera fails to capture informative images and computer vision system is not able to perform calculations normally. The background color of computer vision results turns to red indicating low-reliability condition and the estimated interface level remains unchanged. Over this period, the sensor fusion continuously provides reliable estimation of interface level.

CONCLUSION
A sensor fusion strategy is proposed and integrated with the image processing based computer vision system for froth-middlings interface level estimation. Under normal conditions, the reliable computer vision results are utilized to recursively calibrate the fusion parameters of the existing sensors using the Kalman filter algorithm. Under abnormal scenarios, instead of freezing interface level estimate, the latest calibrated fusion parameters and newly received measurements from other sensors are used to provide reliable interface level estimation results. Through a number of tests, the sensor fusion results are proved to be reliable alternative estimates in abnormal conditions of computer vision system. The FAT concludes that the sensor fusion and computer vision integrated system is able to provide more robust and reliable interface level estimation and the integrated system is implemented in the industry. Besides the PSV system, the proposed sensor fusion approach can be applied to general system that consists of multiple sensors and has high-accuracy but occasionally unavailable or unreliable measurements to improve overall estimation performance.