{'fraction_non_alphanumeric': 0.02439243117209219, 'fraction_numerical': 0.008519415299076316, 'mean_word_length': 4.72778633795583, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
NLOS Satellite Detection Using Fish-Eye Camera and Semantic Segmentation for Improving GNSS Positioning Accuracy in Urban Area

In this paper, for the GNSS (Global Navigation Satellite System) positioning, we propose a method to distinguish non-line-of-sight (NLOS) signals from line-ofsight (LOS) signals by utilizing the satellite geometry and the image data of the fish-eye view of the zenith direction at the antenna position. The NLOS signal is the diffracted signal by buildings or structures, and it greatly degrades the positioning accuracy. By applying the proposed method, we can exclude the NLOS signal and can improve the positioning results in urban areas. In recent years, image recognition using deep learning has developed rapidly. We use the semantic segmentation method using deep learning for segmentation distinguishing sky area from obstacle area.


Introduction
GNSS (Global Navigation Satellite System) is widely used because it can perform highly accurate positioning. However, the accuracy can be easily degraded in places where radio waves are difficult to reach such as urban areas and mountainous areas and so on. GNSS signals tend to reflect on glass, metals, wet surfaces. In urban areas, in addition to signals received directly from satellites, the GNSS receiver may also receive signals reflected on buildings, walls, grounds, etc. This signal is called "multipath signal". Especially multipath signals received from NLOS (Non Line Of Sight) satellites, which are obstructed by obstacles and can not be directly received, are not perfectly corrected even if multipath mitigation techniques are applied [1]. In such case, the positioning accuracy in urban areas is greatly degraded. In this research, we aim to improve the positioning accuracy by estimating the NLOS satellites and excluding them from the positioning calculation.  First, we set omnidirectional fish-eye camera with optical axis aligned to the zenith and, divide the sky and obstacles into regions by using image processing for the images taken by the fish-eye camera. Next, we calculate the elevation angle and the azimuth angle using the satellite position and the approximated receiver Proceedings of the 50th ISCIE International Symposium on Stochastic Systems Theory and Its Applications Kyoto, Nov. [1][2]2018 position calculated from the data obtained from the satellite and plot the satellite position on the image after region division while taking the camera projection method into consideration. Finally, the satellite in the obstacle region is estimated as the NLOS satellite and excluded from the positioning calculation. Fig. 3 shows an example of the detected NLOS Satellites. The left figure in Fig. 3 shows the satellites' positions with satellite identifier such as "G32". In the right figure, satellites G12, G10, R7 and G29 are detected as the NLOS satellites.

FCN
In this research, FCN (Fully Convolution Networks) [2] which is a method of semantic segmentation is used for discrimination of sky area. FCN is an extension of the classification method CNN (Convolutional Neural Network) to the pixel level. This method can be efficiently applied when there exist disturbances such as cloud movement and car body shaking, and it is possible to discriminate the sky region even in such situations, so we consider that the proposed method can discriminate the sky region more accurately in various circumstances. The model applied in this research is fcn8s with the highest accuracy among the three methods described in the paper [2]. FCN has a structure in which convolution layer and pooling layer are alternately arranged. Formula of convolution layer [3] is as follows: where (m, n, k) is the feature map (z l ) size at l layer, (p, q, k) is the kernel (w) size, b is the bias and k ′ is the number of kernel and bias channel. Formula of pooling layer [3] is as follows: where (m, n, k) is the output (z l+1 ) size and R m,n is the rectangular area corresponding to the output destination (m, n). We update the kernel and bias values with learning so that the FCN s output matches the teacher data. However, it is necessary to set the size and number of kernels arbitrarily. Table 1 shows the model and the parameters used in this research. We used the fcn8s, and the parameters in the model are those of VGG16 [4] which is a representative model of the classification in image recognition. We create teacher data using index color with reference to Visual Object Classes Challenge 2012 [5]. Fig.  5 shows an example of teacher data. The right figure in Fig. 5 shows that the red area is the sky area, and the black area is the obstacle area. The teacher image created with index colors is composed of the color map and corresponding indexes, therefore there is an advantage that it is easy to read the index data for classification.

Projective Transformation
In this research, it is necessary to consider the projection method of the camera when we plot the satellite on the image. The lens of the fisheye camera is an orthographic projection therefore we need to transform from equidistant projection to orthogonal projection for plotting. Equidistant projection has a characteristic that the distance from the center and the incident angle are proportional. Fig. 6 shows each projection method. The conversion formula is as follows: where x ort and x equ are expression of orthographic projection and equidistance projection of fisheye lens and θ [rad] is elevation angle of satellite from receiver (0 ≤ θ ≤ π 2 ). Eq. (5) gives the ratio of the distance to the location after projective transformation from the center scaled with a minimum value of 0 and a maximum value of 1. Fig. 7 shows how to calculate the coordinates. We change the scale so that the maximum value becomes the radius of shootable range of the fish eye camera and obtain the coordinates from the azimuth. Coordinate transformation is given by using Eq. (5) as follows: where u and v are uv coordinates with the center of the image as the origin, r is radius of the circle in the shootable range of the fisheye camera and ϕ [rad] is azimuth of satellite from receiver (0 ≤ ϕ < 2π).

Semantic Segmentation
In this experiment, the model is learned by using the sky image and teacher data, and we examine the output accuracy of the inference result. Table 2 shows the experimental conditions. , and expanded these data. We rotated and reversed the images at random 25 times for data augmentation. We set the number of epoch to 100 so that it can be learned sufficiently. Fig. 8, 9 and 10 show the inference result. The left figures in Fig. 8, 9 and 10 show the unlearned images, and the right figures show the inference results. The white area is the sky area, and the black area is the obstacle area.  Fig. 8 and 9, it can be seen that we can infer the sky area with high accuracy. Fig. 10 is an example of failure due to insufficient teacher data of windows reflecting the sky and the sky that pass through the windows of building. It is considered that the accuracy can be improved by increasing teacher data.

Positioning
In this experiment, we estimate NLOS satellites using inference images obtained by semantic segmentation and exclude them from the positioning calculation. Since the purpose of this experiment is to evaluate the accuracy improvement when excluding NLOS satellites, we use single point positioning and least squares estimation which is the simplest positioning method. Single point positioning is a positioning technique that receives a signal from a satellite with a single receiver and estimates the coordinates of the receiver from the geometric distance between each satellite and the receiver. The geometric distance can be measured from pseudo distance data called C/A code pseudo range and the broadcast ephemeris [6]. Table 3 shows the experimental condition. We acquired the data of 450 epochs at 1 second interval in a stationary state with the FLEX6 receiver made by NovAtel and GPS-703 antenna made by NovAtel. Fig. 11 shows the experiment location and antenna position. In order to reproduce the environment in the urban area, the experiment was conducted on the road nearby BIO LINK which is a high-rise building in Ritsumeikan University BKC.  (7) which is a characteristic of the fisheye lens. We inverted the image horizontally in order to match east and west because we took the image in the zenith direction. As a result, the satellite plotted in the obstacle area is estimated as an NLOS satellite. In this experiment, when the NLOS satellite is excluded and the number of satellites used becomes less than 5, we excluded NLOS satellites with low elevation so that there are 5 used satellites instead of excluding all NLOS satellites, considering that the positioning accuracy sharply decreases when there are few satellites used. The yellow pin is the position obtained by the relative positioning method [6] and it is assumed to be true position in evaluating the positioning results in this experiment. The relative positioning is a method that uses two or more receivers and estimates a baseline vector, which is a relative position between each receiver. By using the true value for one of the receiver coordinates, we can perform very accurate positioning. For this reason, we used the reference station data and daily coordinate values (F3 solution) provided by the Geospatial Information Authority of Japan for one receiver [7] [8].
From this result, it can be observed that the coordinates which were scattered in northwest area by the conventional method are gathered around the provisional true value by the proposed method. Fig. 15, 16 and 17 show the ENU error and satellites visibility with each method.   Table 4 shows the statistics of positioning error. From these results, it can be considered that the positioning accuracy in North and Upper direction is improved by excluding NLOS satellites PRN 5 and 12, whereas in the East direction, there is no great improvement in positioning accuracy. It is the influence that the NLOS satellites exist in the south direction. Therefore, we consider that the pseudo-range data from the NLOS satellites contain large multipath errors, consequently they caused the large position errors in the North-South direction. From the above, we confirmed improvement of positioning accuracy by excluding NLOS satellite in SPP (Single Point Positioning).

Conclusion
In this paper, we proposed the NLOS satellite estimation method using a fisheye camera. The FCN used in the sky area segmentation made it possible to infer highly accurate despite learning data a little. It is expected that more accurate inference can be performed by increasing learning data. In addition, we could improve the positioning accuracy by excluding NLOS satellite in SPP (Single Point Positioning). Since there is a problem that positioning can not be performed when the used satellite has not reached the required minimum number of satellites when excluding NLOS satellites, we will also consider not only excluding NLOS satellites but also methods to correct them in the future work.