{'fraction_non_alphanumeric': 0.032355215497944734, 'fraction_numerical': 0.006552439868137235, 'mean_word_length': 4.405191377034756, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Stereo Plane SLAM Based on Intersecting Lines

Plane feature is a kind of stable landmark to reduce drift error in SLAM system. It is easy and fast to extract planes from dense point cloud, which is commonly acquired from RGB-D camera or lidar. But for stereo camera, it is hard to compute dense point cloud accurately and efficiently. In this paper, we propose a novel method to compute plane parameters using intersecting lines which are extracted from the stereo image. The plane features commonly exist on the surface of man-made objects and structure, which have regular shape and straight edge lines. In 3D space, two intersecting lines can determine such a plane. Thus we extract line segments from both stereo left and right image. By stereo matching, we compute the endpoints and line directions in 3D space, and then the planes from two intersecting lines. We discard those inaccurate plane features in the frame tracking. Adding such plane features in stereo SLAM system reduces the drift error and refines the performance. We test our proposed system on public datasets and demonstrate its robust and accurate estimation results, compared with state-of-the-art SLAM systems. To benefit the research of plane-based SLAM, we release our codes at https://github.com/fishmarch/Stereo-Plane-SLAM.


I. INTRODUCTION
Simultaneous localization and mapping (SLAM) is a fundamental problem for various applications, including robots, driverless cars, and augmented reality (AR). Thanks to the increasingly powerful capability of graphics processing, cameras have been widely used and visual SLAM has developed rapidly in the past decade [1].
For visual SLAM, point is the most commonly used feature to track camera poses and build environmental models. Some existing point-based SLAM systems [2], [3] have achieved accurate and robust estimation results. Except for points, lines [4] and planes [5] are also explored to improve SLAM performance in recent years. It is proven that lines and planes are helpful to build more robust and accurate SLAM systems, especially in indoor environments. Line features are commonly extracted using the Line Segment Detector [6] from images. Line segments can be matched by LBD descriptor [7] in frame tracking, but they suffer from the problem of occluding and endpoint variance. Besides, the parameterization of line segments [8] is complicated.
Compared with line features, planes are commonly more robust features in SLAM because of their simple and accurate data association. Planes can even be matched in frames of a large distance, which helps reduce drift errors. Besides, planes are helpful to reflect real scene structures. Plane features are commonly extracted from a dense point cloud [9], generated by a RGB-D camera or a 3D lidar. In stereo images, however, it is not easy to get a dense point cloud accurately or efficiently. The depth of a point can be estimated by matching corresponding pixels in stereo images. But matching all pixels becomes a tough task. Some traditional stereo matching methods [10] utilize low-level features of image patches to match pixels. They run fast but suffer from low quality. Recently, stereo matching algorithms based on deep learning have achieved remarkable performance [11]. But these methods are slow and need high-cost GPU.
This paper proposes a novel method to compute plane features from stereo images. Plane features are commonly extracted from the surface of man-made objects and structures. These planes commonly have regular shapes and straight edge lines. In three dimensions, two intersecting lines determine a unique plane. Therefore, it is reasonable and possible to compute plane features from lines. 3D lines can be computed from stereo images by stereo matching [4]. An example of computing plane features is shown in Fig. 1, the original scene is also shown at the top-left corner. We also provide a video at https://youtu.be/3VWF-JJU9T8.
In summary, our contributions are as follows: • A novel method to compute plane features from stereo images based on intersecting lines. • A stereo SLAM system using extracted points and computed planes. • Evaluated on public datasets, our system gets accurate estimation results and achieves state-of-the-art performance. In the following, we first introduce the related work in Sec. II, then explain the method to compute plane features in Sec. III, followed by the introduction of the whole system in Sec. IV. In the end, we show our experimental results in Sec. V.
II. RELATED WORK SLAM is well studied and different methods have been proposed in recent years. Many SLAM systems are commonly based on point features and build a global map consisting of 3D points. ORB-SLAM [2] tracks ORB features and uses re-projection error to estimate camera poses. In contrast, direct methods [12] use photometric error to track camera poses. Point features can be used to build a sparse map [13], semi-dense map [14] and even dense map [15]. In these methods, it is easy to extract and compute point features, and they work well in most scenes.
In recent years, researchers also utilize line and plane features to refine the performance of SLAM systems, especially for indoor environments. In a SLAM system, line and plane features work like a filter to reduce the measurement errors, and perform well even in some low-texture scenes. Pumarola et al. [16] extract both point and line features from monocular images and improve SLAM performance for low-texture scenes. Zhang et al. [17] propose a 3D linebased SLAM system using a stereo camera and exhibit its better reconstruction performance. PL-SLAM [4] is built based on ORB-SLAM, and also leverages both points and line segments. Qian et al. [18] extend the work of PL-SLAM using Bags of Point and Line Word and release their codes.
Compared with line features, planes are more accurate and robust landmarks. Taguchi et al. [19] present a framework for registration combining points and planes. CPA-SLAM [20] proposes a novel formulation to track camera poses using global planes in the expectation-maximization (EM) framework. Kaess et al. [5] introduce a minimal representation for infinite planes which is suitable for the least-squares estimation without encountering singularities. [21] explores both plane and plane edges to increase plane observations and constraints. These plane-based SLAM systems also achieve more robust and accurate estimation results compared with point-based methods, especially in low-texture scenes. All of these systems are based on RGB-D cameras, from which planes are easily extracted. Our work utilizes some ideas from line-based SLAM, and extends plane-based SLAM to using stereo cameras.

III. PLANE FEATURES FROM INTERSECTING LINES
This section is to introduce the method of computing plane features. We first extract line segments from both left and right images of a stereo camera. By matching line segments and their endpoints, we compute 3D positions of endpoints  and lines' direction vectors. Then we check their position to find intersecting lines. Finally, we compute plane parameters.

A. Notations
We denote a plane feature as π = n n n , d , where n n n = (n x , n y , n z ) is the unit normal vector representing the plane's orientation and d is the distance of the plane from the origin. We use the commonly used form T T T cw ∈ SE(3) to represent a camera pose and p p p = (x, y, z, 1) to represent a 3D point. Therefore, T T T cw p p p w transforms a point from the world to the camera coordinate system and T T T − cw π w transforms a plane from the world to the camera coordinate system.
For lines, we only record their endpoints (p p p s , p p p e ) and unit direction vectors n n n l , which are enough to compute plane features.

B. Line Detection and Computation
A frame from a stereo camera consists of a left image I l and a right image I r . We use the Line Segment Detector [6] to extract line segments from both I l and I r , and match them by the LBD descriptor [7]. The line matching is accurate and robust enough in one stereo frame. As shown in Fig.  2(a), the line segments are drawn in different colors, and the matched line segments are the same color in I l and I r .
For every matched line segment in the left image I l , we find corresponding points of its endpoints in the right image I r based on the fact that their row positions remain the same in I l and I r when using a parallel stereo camera. As shown in Fig. 2(b), matched endpoints are connected by transverse lines.
From the stereo matching of endpoints, we compute their 3D positions p p p based on disparities ∆u. A line's direction vector n n n l is also defined by its two endpoints.

C. Plane Computation
Before computing plane features, we need to check the relationship of lines. In three dimensions, two intersecting or parallel lines are on the same plane. For parallel lines, however, it is difficult to judge whether they are extracted from a real plane, and the planes computed from them are prone to bring large errors. Therefore, we only compute planes from intersecting lines.
To find intersecting lines quickly, we find the lines meeting the following conditions: • The angle between two lines is larger than the threshold (10 • in our experiments) • The distance between their central points is smaller than the length of the line. • The four endpoints of these two lines lie on the same plane. The central points p p p c in the second condition are computed from lines' endpoints p p p s and p p p e . From the first two conditions, we actually find unparallel lines that are close with each other. We compute the plane's normal vector by the cross product of the lines' direction vectors, n n n π = n n n li × n n n l j (1) Using the plane's normal n n n π and four endpoints p p p k (k = 1, 2, 3, 4), we then compute four different plane coefficients d k , d k = −n n n π · p kx , p ky , p kz (2) The distance among them is: If D is smaller than the threshold (5 cm in our experiments), these two lines meet the third condition and the plane coefficients π = n n n π ,d k are also computed,d k here is the arithmetic average of d k . Sometimes the computed planes may not be real planes, such as the plane from the lines of a doorframe. But such planes are also stable enough and provide accurate constraints, and therefore we treat them as real planes.
Under these conditions, we compute as many planes as possible at first. We will validate the computed planes in the frame tracking and label inaccurate planes as invalid, which is described in Sec. IV-E.
The features extracted and computed from the images of Fig. 2 is shown in Fig. 1. The black points are feature points extracted from the images. The red lines are the line segments extracted from the images, whose 3D positions are computed from the matched endpoints. Notice that we do not use these line segments in our SLAM system, and we draw them in the figure just to show the process of computing plane features. We draw the plane features by expanding corresponding intersecting lines and they are drawn in different colors. The red plane in the middle of the figure seems not correct because of errors from line segments, and we label this plane as invalid in the validation. But other planes are correct and are used as valid landmarks in our SLAM system.

IV. SLAM SYSTEM BASED ON COMPUTED PLANES
Points and planes are both used as landmarks and optimized in our SLAM system. We build our system based on the publicly available ORB-SLAM stereo version [2], which includes feature tracking and bundle adjustment optimization [22].

A. System Overview
The pipeline of our proposed SLAM system is illustrated in Fig. 3. It can be divided to three parts, frame processing, pose tracking and mapping. We do not add loop closure now, which will be a part of our future work.
In stereo frame processing, we extract feature points and line segments from both left and right images, and match these features based on descriptors. Then we can compute plane features using the method described in Sec. III.
In pose tracking, every camera pose is estimated based on matched valid features. The camera pose is firstly estimated from the last keyframe and then optimized in the local map.
In mapping, points and planes are constructed and saved in the map. To achieve more accurate estimation, a local map optimization is performed.

B. Optimization Formulation
SLAM is commonly formulated as a nonlinear least squares optimization problem [1], and the bundle adjustment (BA) is commonly used for point features [2]. Like points, we also design the optimization formulation for plane features. In our SLAM system, we denote a set of camera poses, point features and plane features as C = {c i }, P = {p j }, Π = {π k } respectively, then the optimization problem can be formulated as: e (c, p), e (c, π) represent the camera-point and camera-plane measurement errors respectively. x x x 2 Σ is the Mahalanobis distance, which equals x x x Σ −1 x x x. Σ is the corresponding covariance matrix, it is set based on the measurement uncertainty (such as 2 • for plane angles in our experiments).
The optimization problem can be solved using Levenberg-Marquardt or Gauss-Newton implemented in g2o [23].

C. Measurement Error 1) Camera-Point Error:
We use the standard re-projection error for the camera-point measurement in our system: e e e cp (T T T cw , p p p w ) = u u u c − ρ (T T T cw p p p w ) (5) Here T T T cw is the camera pose, p p p w is the point in the world coordinate system, u u u c is the corresponding image pixel, and ρ is the camera model to project 3D points onto the images. In optimization, the camera pose T T T cw is mapped to Lie algebra ξ ξ ξ ∈ se (3) to avoid extra constraints [24]. The computation of the corresponding Jacobian matrix can also be found in [24].
2) Camera-Plane Error: As a 3D plane has only three degrees of freedom, π = n n n , d is over-parameterization. Therefore, it requires extra constraints to ensure the unit length of planes' normal vectors, adding additional computation in optimization. To overcome this problem, we follow the work in [20] using the minimal parameterization of planes τ = (φ , ψ, d) in optimization, where φ and ψ are the azimuth and elevation angles of the normal vectors respectively: τ = q(π) = φ = arctan n y n x , ψ = arcsin n z , d Then we define the measurement error using the minimal parameterization: e e e cl (T T T cw , π w ) = q (π c ) − q T T T − cw π w (7) Here π w is the plane parameter in the world coordinate system and π c is the observation of planes in the camera coordinate system. The camera-plane error defines the differences between the plane landmark and its corresponding observation in the camera coordinate system. The computation of the Jacobian matrix can be found in our previous work [21].

D. Data Association
We try to match every observation of points and planes with the landmarks in the map. This process is defined as data association. Robust data association is also necessary for accurate estimation results. Point features are commonly matched by their descriptors.
To match plane features, previous works [5], [20] depend on the plane parameters n n n and d π directly. This method is simple and works well for small scenes. But it needs accurate estimation of camera poses, and d π may fluctuate largely because of errors. To refine the data association of planes, we use the distances of endpoints instead.
After computing planes from intersecting lines, we save the endpoints of the lines. To match plane features, we compute the average distanced ep from the endpoints to plane landmarks. Unlike d π , the fluctuation ofd ep is relatively small. Ifd ep is smaller than the threshold (6 cm in our experiments) and the angle between plane normal vectors is also smaller than the threshold (12 • in our experiments), the plane landmark is matched with the corresponding plane observation.

E. Plane Validation
We use the same method to select keyframes as in ORB-SLAM [2]. From the keyframe, every unmatched plane feature is used to create a new plane landmark. But these plane landmarks may be inaccurate, and therefore they are labelled invalid. The planes are changed to be valid only if they are observed in sufficient frames (3 keyframes in our experiments). Only valid plane features are used to estimate camera poses and added into the optimization framework (Eq. (4)).

V. EVALUATION
In this section, we evaluate our proposed SLAM system in two popular public datasets: the EuRoC dataset [25] and the KITTI vision benchmark [26]. The two datasets both provide stereo images. All experiments run on a laptop computer with i7-7700HQ 2.80 GHz CPU, 16GB RAM, without GPU.
We compare our proposed system with other state-of-theart stereo SLAM systems. ORB-SLAM2 [2] is a popular point-based visual SLAM system and it has a stereo camera implementation. We also compare our system with a linebased SLAM system, which utilizes the line segments directly. [18] provides a SLAM system based on point and line features, and it is also built on ORB-SLAM2 and achieves state-of-the-art performance. The two SLAM systems both provide open-source codes on the Internet. Our work mainly focuses on the front-end, thus we switch off loop closure to compare drifts in our experiments.
For implementation, our system augments the stereo variant of ORB-SLAM2. We rely on the underlying ORB-SLAM2 for the point extraction and matching, and the methods to maintain a local map. The focus of our implementation is on the line segment extraction, plane computation, plane matching, and camera pose estimation using both point and plane features. In the end, we construct a global map consisting of both points and planes. To help understand the details of our system, we release our codes at https://github.com/fishmarch/Stereo-Plane-SLAM.

A. EuRoC Dataset
The EuRoC dataset contains stereo sequences recorded from a micro aerial vehicle flying around in three different indoor environments, including an industrial machine hall and two Vicon rooms. These scenes contain man-made objects and structures, which are easy to extract line segments and compute plane features. The sequences are classified as easy, medium, and difficult according to the speed, illumination, and scene texture. The dataset also provides ground truth from the laser tracking system and the motion capture system. Table I shows the comparison of estimation results of different SLAM systems. Here we use the absolute translation root mean square error (RMSE) to evaluate the estimation results. The smallest error for each sequence is labelled as a bold number. It is clear that our system outperforms the stereo ORB-SLAM2 in these sequences. The computed plane features bring more constraints to estimate camera poses, and these constrains are reasonable in such indoor environments. The line-based SLAM system also performs better than ORB-SLAM2 in these sequences, and gets comparable results with ours. Although using line segments directly adds more information to the SLAM system, some line segments are not accurate and the data association is difficult when the camera view changes, as shown in Fig. 4. A single line in the scene may be constructed as several segments because of the failure of data association. By computing planes from line segments, our system may lose some information but filters out inaccurate line segments, and the data association of planes is easier between frames. The estimation results of our system are better in industrial machine hall sequences(MH*). We find the objects and structure in the industrial machine hall have clear and sharp edges, which benefits the line extraction and plane calculation. But in the Vicon room (V1* and V2*), there are more errors of the lines extracted from the soft cushions and curtains.
An example of the built map from our system is shown in Fig. 5. The map consists of points, planes and camera poses. The plane features enrich the map and help to reflect the real scene structures clearly. It demonstrates the plane features are computed from the main objects and structure, such as walls, cushions and etc. The planes from the wall and the ground are easily matched even in frames of a large distance. Although we try to compute accurate planes and filter out those with large errors, some inaccurate planes still

B. KITTI Dataset
The KITTI dataset contains stereo sequences recorded from an autonomous driving platform driving in urban and highway environments. The ground truth is given by the GPS/IMU localization unit. Although our SLAM system is more suitable for indoor environments, the planes can also be computed from man-made structures (such as houses and roads) in these sequences. Table II shows the comparison of estimation results in the KITTI dataset. We again calculate the RMSE of these sequences and label the smallest errors as bold numbers. Our system also gets better estimation results in most of the sequences. The performance of ORB-SLAM using only point features is good enough, and our system only gets similar results in some sequences. This is because the camera moves much faster in these sequences, and thus planes are only matched in a few frames and fewer valid planes are created. Fig. 6 shows the examples of the estimated trajectories compared with the ground truth and our results are closer to the ground truth.
The line-based SLAM system also performs better in some sequences. But in other sequences, it gets even worse results, such as the Sequence 01, 02, and 07. This is because the extracted lines have large errors and are difficult to match when the camera moves quickly.
A part of the built map in the Sequence 00 is shown in Fig.  7. The plane features are computed from roads and walls. It  is clear the drift error is small.

C. Time Performance
We record the average processing time of the main parts of the proposed system, as shown in Table III. In the table, feature processing includes point extraction, line extraction, stereo matching, and plane computation; feature tracking includes landmark matching and camera pose estimation. Frame tracking is the whole process for tracking a new coming stereo frame. The most time-consuming part is the feature processing, specifically point and line extraction. Because the images from KITTI sequences are larger, feature processing needs more time.
We also record the average processing time of the line-based SLAM system [18]. The average time for its frame tracking are 81.8247ms in the EuRoC sequences and 105.347ms in the KITTI sequences. It is clear that our system is faster than the SLAM system using line segments directly.
In the line-based system, many line segments are built with large errors, and they are hard to be matched correctly, as shown in Fig. 4. But all of these features are added into the optimization function, and therefore it needs more time to estimate the optimal results.
VI. CONCLUSIONS We have presented a novel method to compute plane features from stereo images for visual SLAM. Many previous works [5], [20], [21] have demonstrated the benefits of adding plane features in SLAM systems, but most of them are for RGB-D cameras. In this paper, we compute planes from stereo images instead, based on the truth that two intersecting lines determine a plane. After further validation, we add computed planes into our stereo SLAM system. We have presented our experimental results in two famous public datasets and demonstrated the accuracy of our system.
From the experimental results, it is clear that our system outperforms the state-of-the-art point-based SLAM system. Compared with the line-based SLAM system, our system also gets comparable results. The plane computing filters out inaccurate line segments and adds stable constraints to estimate camera poses. The constructed map consists of both points and planes, which reflect the real scene structures. The structured environments with man-made objects are ideal working scenes for our system.
From the constructed maps, we notice some inaccurate plane features still exist and they bring a big challenge to the data association. In the future, we would like to refine the plane computing and validation methods to get more accurate and robust plane features. In addition, we also need a more robust data association algorithm, removing the influence of estimation errors. We will also add loop closure into our system.