{'fraction_non_alphanumeric': 0.026549958231939855, 'fraction_numerical': 0.011331856317873097, 'mean_word_length': 4.75423197492163, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LiveChess2FEN: a Framework for Classifying Chess Pieces based on CNNs

Automatic digitization of chess games using computer vision is a significant technological challenge. This problem is of much interest for tournament organizers and amateur or professional players to broadcast their over-the-board (OTB) games online or analyze them using chess engines. Previous work has shown promising results, but the recognition accuracy and the latency of state-of-the-art techniques still need further enhancements to allow their practical and affordable deployment. We have investigated how to implement them on an Nvidia Jetson Nano single-board computer effectively. Our first contribution has been accelerating the chessboard's detection algorithm. Subsequently, we have analyzed different Convolutional Neural Networks for chess piece classification and how to map them efficiently on our embedded platform. Notably, we have implemented a functional framework that automatically digitizes a chess position from an image in less than 1 second, with 92% accuracy when classifying the pieces and 95% when detecting the board.


Introduction
The recent breakthroughs in Deep Neural Networks [23,22,18,38] have provided an astonishing advance in the application of image classification algorithms. Many Convolutional Neural Networks (CNNs) have been employed for this task, such as MobileNet [33], Xception [6] or NASNet [42]. These CNNs have been continually studied and improved to perform well even for large-scale image classification [22].
Nevertheless, the efficient recognition of chess pieces and chessboards is still an unsolved computer vision problem [8,11]. Its solution will benefit experienced players who want to study and train using chess engines and other specialized software programs but prefer to work with a physical chessboard. It will also be useful for chess tournament organizers who want to broadcast OTB games online or for amateur players who want to share their OTB games with friends.
Specialized chess sets (electronic boards) can efficiently perform this digitization task, but they are expensive. As an alternative, previous work has explored affordable solutions based exclusively on computer-vision [8,14]. Overall, the problem can be broken down into two main parts. The first step is to recognize the chessboard and its orientation, and then identify the chess pieces and their precise position afterwards.
In this paper, we present LiveChess2FEN, a framework that tries to fulfill the expectations raised by that state-of-the-art solutions [8,14]. We have focused on optimizing the recognition process and reduce its latency as much as possible. Note that in addition to accuracy, some of the envisioned applications require very low latencies. This is the case of broadcasting OTB games online where players can make moves even in fractions of a second, especially in game modes known as "Bullet" or "Blitz".
Remarkably, LiveChess2FEN can identify all the chess pieces of a given position in less than a second, outperforming the recognition latency of previous work by a factor of five [8]. This improvement has been possible thanks to the deployment of CNNs on top of an Nvidia Jetson Nano single-board computer and the implementation of additional optimizations that exploit domain-specific information related to the chess rules. In this manner, we avoid the use of computationally-expensive chess engines such as Stockfish [36].
The rest of the paper is organized as follows: Section 2 describes the state of the art in chessboard and pieces recognition. Section 3 introduces FEN notation, the final output of the LiveChess2FEN framework, which is described in Section 4. Section 5 details the chosen hardware platform and the optimizations made. Section 6 reports the results of each of the digitization steps and the performance achieved for the full digitization. Finally, Section 7 outlines the conclusions and future lines of work.

Related work
Specialized boards that physically detect the pieces are a hardware solution for automatic chess digitization [37,12]. However, these boards are expensive and difficult to deploy in many areas. For example, a set of DGT chessboard and pieces (brand used in official tournaments) costs between e500 and e1000 [13].
Other alternatives are those provided by robots that move the pieces on a board, such as [26] or more recently [5] and [21]. These robots are based on positioning a zenith camera over the board and detecting the differentials between one movement and the next. A drawback of this approach is that it is necessary to start from a known initial position. A board with a generic position could not be digitized this way. In addition, errors should be taken into account because they could add up in each new play.
The solutions offered by computer vision are an alternative to consider since they provide cheaper and increasingly accurate systems. Furthermore, there are several platforms with enough computational power to perform these tasks at an affordable cost. For instance, the Nvidia Jetson Nano costs around e110 [28] and does not just stick to one function, it could be reused for other tasks. Other alternatives include the Intel Neural Compute Stick 2 [25] or Google's Coral boards [17]. These computer vision procedures are based on combining and adapting transformations and detectors already known and used in other fields, such as the Harris corner detector and the Hough transform [10]. Many of the methods often assume significant simplifications, such as determining the exact position of the camera, using boards designed explicitly with markers to aid in corner detection, or directly through user interaction [14]. However, some generic solutions that overcome these restrictions already exist.
For example, some methods allow us to classify occurrences of various objects in arbitrary places using CNNs [16]. However, they do not have the precision required to obtain their exact location. The authors of [2] describe a method for object detection that also uses CNNs trained through weak supervision. That is, it is enough to have labeled images of the objects to train the network, without the need to provide the exact location of the object in each training image. The problem with this approach is that it takes many iterations to locate an object precisely, which does not make it very practical in situations that require fast responses.
The authors of [8] propose a method for chessboard detection that is robust against light conditions and the angle from which the images are taken. In addition, it works with most styles of boards and it overcomes many of the weaknesses that we have been discussing. It is an iterative process in which the location of the board is refined in several phases. The authors obtain 99.5% accuracy in detecting the intersections of the center board grid and find the full location of the chessboard accurately 95% of the time.
Regarding piece classification, once the board has been located, in [14], a method is proposed that is based on Support Vector Machine (SVM). This is trained on the features extracted by SIFT [24], thus achieving 85% accuracy when classifying the pieces. In [11] a method for training CNNs from artificially generated 3D images is introduced. Authors obtain 97% accuracy, although these are computer-generated scenarios, and no testing with real chessboard and pieces is done.
In [40] authors introduce an alternative to CNNs, oriented chamfer matching. They obtain comparable results to those of CNNs, but with a lower training set in exchange for fixing the piece types, as they rely heavily on template matching. In [8] authors claim to achieve 95% accuracy classifying chess pieces using a custom CNN, which they enhance by clustering similar pieces, taking into account their height and area, and using a game engine (Stockfish) to obtain the probability of particular positions. The code for these enhancements has not been released, so we could not analyze this particular method.
In this paper, we leverage the approach in [8] to detect the board. Several optimizations have been performed to accelerate its execution, as described above.
As for the piece classification in an arbitrary snapshot, different CNNs have been studied and mapped onto an Nvidia Jetson Nano board. Finally, all the different scenarios are tested and put together to form the LiveChess2FEN framework.
3 Preliminaries: FEN notation FEN notation [15] is a string representation of a position of the board. This will be the final output of our digitization, as it can be imported directly into chess engines or other computer programs to visualize a digital chessboard.
Since only a snapshot of the game will be available, only the pieces' position at a particular moment in time will be known. Not the player whose turn it is to move or if there is a possibility of castling for example (factors that are taken into account in the full FEN standard). Thus, the output string is a series of eight blocks of alphanumeric characters representing each row of the board separated by the / character. For instance, the initial position of a game is encoded as:

LiveChess2FEN
LiveChess2FEN is a functional framework capable of executing the entire process of digitizing a chess game photo in real-time, making all the necessary calculations on specialized hardware. This has been carried out having in mind its use in amateur games or in tournaments, taking the photos from the side of the board each time a player presses the clock ( Figure 1). We show a schematic of the camera position in Figure 2. The full digitization process is done in two major steps. The first step is to locate the chessboard in the input image. Subsequently, the algorithm must classify each of the squares into the corresponding chess piece. An overview of the full process is shown in Figure 3.

Board detection
Locating a chessboard with pieces hiding part of its grid is a complex machine vision problem. Nevertheless, this step must be extremely precise when locating the board's four corners since these coordinates are used to split it into its 64 squares. With this information, we can crop the empty squares and the chess pieces from the original image and locate their positions automatically. Reducing errors in this step is essential to classify the chess pieces correctly afterward. Authors propose in [8] a fast iterative process, which starting from a photo taken from the surroundings of a board, is able to locate it with enough precision after just a few iterations. Our proposal is based on this previous work and introduces further optimizations to reduce the execution cost and thus the latency, as shown in Section 5.
In situations where the board and camera are going to stay still, the framework can take advantage of this fact to avoid recalculating the location of the board each time. In these cases, it must store the coordinates of the corners calculated for the previous image, check that the board is still in the same place and proceed to separate the squares directly.
We have implemented an algorithm that can quickly check if the board is still in the same location. For this, the algorithm employs the geometric detector and the neural network used when locating the chessboard grid points in the board detection phase. If the 49 corners of the central 6 × 6 square of the board are still in place, the previous information that is stored is still correct. When counting the corners that are part of the grid, a tolerance margin must be taken since, for instance, some points may be occluded by a piece. Experimenting on different test images, we have concluded that spotting 20 of the 49 points is enough to confirm that the board is still in the same place (Algorithm 1).

Chess piece classification
Once the board's location in the image is known, it must be separated into its individual squares to classify them and obtain the final result. This is done by simply dividing the image from the last iteration of the chessboard detection, a square, into an 8 × 8 grid.
After dividing the original image into each of the board's squares, the next step is to classify them. Each square can be empty or occupied by a piece of one of the players, so it is necessary to decide which of the 13 classes corresponds to each of the board's 64 squares. Currently, the most widespread and the best algorithms to classify an image into a series of categories are CNNs [32]. Furthermore, these networks can be significantly accelerated, as shown in Section 5.
The final result of the classification is a string that encodes using FEN notation the position extracted from the image, as described in Section 3.
In order to train a deep learning model, a large number of images are necessary. Furthermore, if the network must also classify different types of pieces, it is necessary to introduce this variety in the training data. To train our models with enough variety, two labeled datasets of chess pieces [41] [34] have been put together. Thus, the employed dataset consists of nearly 55000 images of chess pieces.

Algorithm 1: Board location check.
Input: Image containing a board. Candidate corners to continue being part of the central 6 × 6 square of the board. Output: If the board is still in the same location.
if is grid corner then 6 correct corners ← correct corners + 1; The high-level Keras API [7] on top of the TensorFlow library [1] was used to define and train the CNNs with which to test the piece classification. Transfer learning and fine-tuning techniques were used to ease this process. To choose the pre-trained models for our dataset, we leveraged the exhaustive study carried out in [4]. This was especially useful since the authors also tested different configuration options available on the Nvidia Jetson Nano, which has also been our platform of choice for the inference.
Based on these results, we decided to test MobileNetV2 [33], NASNetMobile [42], DenseNet201 [19], Xception [6], AlexNet [22] and SqueezeNet-v1.1 [20]. AlexNet and SqueezeNet-v1.1 have the lowest inference times, especially when executing in batches. MobileNetV2 is a larger model, but it is very versatile, as it can be tuned with an α parameter to control the width of the network. NASNetMobile, DenseNet201 and Xception are more complex and thus slower, but achieve a higher accuracy while still requiring a reasonable amount of resources to run.
The output of the deep learning models is a 13 component vector. Besides using this vector of probabilities, some domain knowledge has been integrated into our flow to improve the piece classification accuracy. In this way, chess rules were introduced to take into account all of the squares at the same time, as opposed to the classification of each square using just the output of the deep learning model, which is agnostic of its surroundings. This process is summarized in Algorithm 2. Firstly, the algorithm finds the two squares with the greatest probability of containing the kings. This ensures that there is exactly one king of each color. Subsequently, all of the empty squares are set since the models detect them with enormous precision. Finally, the rest of the squares are classified, considering that each piece has a maximum number of appearances. In addition, if a player keeps the bishop pair, those bishops must be in squares of different colors 1 .
In order to classify the pieces that are not kings, the program follows the following steps. First, the remaining squares are sorted according to their probability of containing each of the remaining 10 classes (in total, there were 13, but the kings' positions and the empty squares have already been decided). Thus, 10 lists of pairs of pieces and squares are obtained ordered from highest to lowest probability of containing the corresponding piece (the top vertical lists in Figure 4). Afterward, the algorithm iterates by choosing the element at the top of the list with a higher probability (tops vector). If the maximum occurrences of that type of piece have not yet been reached and the square it represents on the board has not yet been filled, it is selected. In any case, the piece is removed from its list of ordered pieces, as it has already been processed ( Figure 4).
In the next iteration, the piece that has the second-highest probability is chosen, and the process is repeated. The algorithm finishes after covering the entire board. In this way, as shown in Section 6, the piece classification accuracy has been increased.

Optimizations and acceleration
The final step and the main contribution of our work has been accelerating the entire framework on a target embedded platform. Our platform of choice is the popular Nvidia Jetson Nano single-board computer [28], which offers 472 GFLOPs of FP16 compute performance.
In addition to a dedicated Nvidia GPU, this system integrates a quad-core ARM CPU capable of performing the sequential computation necessary to detect the chessboards. This type of architecture has been widely used successfully for more than a decade in tasks related to image processing [35] [39].
The neural networks have been defined and trained using Keras on top of Ten-sorFlow. In order to improve the inference execution times, we have leveraged the ONNX (Open Neural Network Exchange) model representation format [31] and its optimizer ONNXRuntime, [27] as well as TensorRT [29], the deep learning inference 1 In our study, we have assumed that, if there were any previous untracked promotions, they have been to a queen. In a promotion, the player chooses any piece except a king or another pawn. However, the occasions when a player does not choose a queen are rare. This assumption was made since no knowledge of previous positions is expected, but it can be completely eliminated if the whole game's history is available. In these cases, it would be possible to precisely know the number and type of pieces each player has.  optimizer developed by Nvidia ( Figure 5).
The execution times of the whole framework have been further reduced by optimizing the bottlenecks spotted in the original flow when detecting the board [8]. The most important improvements were the following: • Accelerating the inference of the neural network used to classify the most complex lattice points of the board's grid. For this purpose, the original model was transformed into the ONNX format, and its execution was optimized with ONNXRuntime.
• Reducing the overhead caused by the calls to the NumPy library [30]. When computing the distance from a point to a line in two dimensions, the general formula (y − x) × (x − z) can be simplified to which can be calculated without the need of complex mathematical libraries.
• Simplifying some calculations when computing the intersections of a set of lines. In cases where the size n of the set is small, the Bentley-Ottmann algorithm [3], which has an asymptotic complexity of O((n + k) log n), where k is the number of intersections, is actually significantly slower than the naive approach, which has an asymptotic complexity of O(n 2 ).

Experiments and results
In this section, the final results are analyzed and compared with the initial results and with other proposals. In addition, the total time that the program takes to execute the complete digitization is studied. Starting by loading the photo of a board and finishing with the calculation of the FEN notation of its position.

Board detection
The modifications introduced in the baseline code [8] do not affect the final precision when detecting the board. Therefore, our flow maintains the same accuracy: 99.5% when detecting the intersections of the board's central grid and accurately finds the location of the board in the image 95% of the times. Regarding the final execution times, a noteworthy improvement has been achieved. The set of 10 photos used in [8] has been employed in order to calculate the runtimes. As the goal is to minimize the latency, we measure the time elapsed since the beginning of the loading of the input photo and until the cropped board image is saved. All the optimizations mentioned in Section 5 have reduced the latency in detecting the board by a factor of four. A speedup of 4.27 has been achieved, reducing the time required from 16.38 to 3.84 seconds per board on average (Table  1). It must also be considered that in [8], authors comment that they get much higher accuracy in exchange for an execution time increase, which in the end is up to two times slower than other alternatives. Hence, we could conjecture that this improved method is twice as fast as those proposed in [10] and [9], as well as being more accurate.  Table 2: Top-1 value and accuracy after including the global domain knowledge into the inference of each of the models.

Piece classification
In this section, we have evaluated the accuracy when classifying pieces by means of the Top-1 value after applying the corresponding CNN model. Moreover, this has been complemented by including the domain-knowledge improvements. To gather these measurements, tests on 5 chessboard photos 2 have been carried out. Each board has between 21 and 32 pieces in various positions drawn from real games.
To avoid overfitting when evaluating the accuracy, these boards contain pieces of a different type from the ones used in the training dataset. In this way, the robustness of each model to changes in the type of the pieces can be verified.
These results are shown in Table 2. It is possible to observe that the accuracy increases between 1 and 4 % on average when including domain information. However, it must be noted that, in some cases, including this knowledge worsens the accuracy slightly due to corner cases. For example, if there are two squares with a very high probability of containing a black king, it may be the case that this piece is located in the square that has a slightly lower probability than the other one. In this case, the domain information algorithm would choose the wrong square to position the king, but the probability vectors would assign a black king to both  squares. Therefore, the former algorithm would fail to guess both of the squares, and the latter would guess correctly one of them. Nevertheless, this global domain knowledge increases the accuracy in most situations and inserts coherence in the results. At all times, they are positions that could occur in a real chess game. Finally, it must be noted that these accuracies remain similar when executing the piece classification models on the optimizers that have been considered.
The execution time is depicted in Table 3. In this table, the best results achieved with each model are displayed. As can be seen, the best inference engine has always been TensorRT with a batch size of 64, except in the NASNetMobile case, where it was not possible to transform the ONNX model. Notably, the original inference times of the models ranged between 5.86 and 28.96 seconds per board, while in our case, these range from 0.46 to 6.04 seconds.
According to Table 3 there is no best solution, so Figure 6 shows the Pareto Front when studying both accuracy and execution time for the aforementioned models and optimizers. Furthermore, the number of parameters of each network is illustrated through circles of different sizes. Four models appear on the Pareto Front, namely: SqueezeNet-v1.1, MobileNetV2(α = 0.5), NASNetMobile and Xception. Among these, Xception and NASNetMobile possess the highest accuracies. However, they require more than 5 and 3 seconds, respectively, to perform the classification of all the squares of the board in the image. Hence, for the scope of this work, SqueezeNet-v1.1 and MobileNetV2(α = 0.5) are the best candidates, as their execution time is far below 1s and their accuracy exceeds 90%.

Full digitization
The full digitization of a chessboard comprises from the moment the system detects that there is a new image to be processed until it finishes processing it and produces the output FEN string. Besides the detection of the board in the initial image and the classification of the pieces, which practically involve the entire execution time, three additional operations are required to complete the whole process.
Firstly, once the chessboard has been detected, the individual squares must be separated. This takes around 80 milliseconds on the Jetson Nano. Subsequently, after obtaining the probability vectors of the network being employed, the actual position has to be inferred. Finally, this information is transformed into the FEN notation. The sum of the time required to execute these two operations is slightly over 10 milliseconds. Therefore, roughly 100 milliseconds must be added to complete the digitization. Table 4 summarizes the execution times of the aforementioned tasks when performing the whole process. As can be observed, when employing SqueezeNet-v1.1 or MobileNetV2(α = 0.5), it is possible to digitize an image in times that range from 3.8s to 5s.
In situations where the camera and the board remain still, we have introduced in Section 4.1 an algorithm to check if the board's location in the image is the same as in previous images. Running this algorithm on the Jetson Nano takes about 150  milliseconds per board. Therefore, when this test returns a positive result, a huge reduction in the total execution time is achieved. In Table 5 the times when this board check returns true are summarized. The slight extra cost added when the test is false is highly compensated. In Table 4 it is shown that the board that is detected the fastest takes 3.30 seconds. Therefore, if this check returned true 1 out of 14 times, these calls would be amortized. As can be seen, this prediction would reduce the total time required to digitize new positions to less than one second. In addition, periodic sampling can be added to capture possible intermediate moves. In this way, cases in which there is, for example, a hand covering part of the board could be avoided. Moreover, sometimes players forget to press the clock. In these cases, this periodic sampling would be necessary in order to capture all the moves.  to perform the tests. Regarding accuracy when classifying the pieces, the investigated method obtains comparable results to those of other proposals. Several CNNs have been tested, reaching a good trade-off between speed and accuracy with SqueezeNet and Mo-bileNetV2. On top of this, several domain-based rules have been considered to optimize accuracy further. In this manner, we have avoided the usage of CPU intensive chess engines as in the literature solutions.
Notably, the training and testing in our experiments have been performed with different chess sets, which highlights the robustness that can be achieved using CNNs. The complete source code of the LiveChess2FEN framework is publicly available with an open-source license in our GitHub repository: https://github. com/davidmallasen/LiveChess2FEN.
Finally, we should note that we have found a barrier to improving the piece classification accuracy while training different CNNs. The simplest models have been able to learn practically all the information available in our dataset, and training more complex models has provided almost no benefit ( Table 2). With this in mind, we believe that part of this problem would be solved by introducing more information into the dataset.