{'fraction_non_alphanumeric': 0.02692151399047951, 'fraction_numerical': 0.009360849878091257, 'mean_word_length': 4.74112647892018, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation

In order to explore robotic grasping in unstructured and dynamic environments, this work addresses the visual perception phase involved in the task. This phase involves the processing of visual data to obtain the location of the object to be grasped, its pose and the points at which the robot`s grippers must make contact to ensure a stable grasp. For this, the Cornell Grasping dataset is used to train a convolutional neural network that, having an image of the robot`s workspace, with a certain object, is able to predict a grasp rectangle that symbolizes the position, orientation and opening of the robot`s grippers before its closing. In addition to this network, which runs in real-time, another one is designed to deal with situations in which the object moves in the environment. Therefore, the second network is trained to perform a visual servo control, ensuring that the object remains in the robot`s field of view. This network predicts the proportional values of the linear and angular velocities that the camera must have so that the object is always in the image processed by the grasp network. The dataset used for training was automatically generated by a Kinova Gen3 manipulator. The robot is also used to evaluate the applicability in real-time and obtain practical results from the designed algorithms. Moreover, the offline results obtained through validation sets are also analyzed and discussed regarding their efficiency and processing speed. The developed controller was able to achieve a millimeter accuracy in the final position considering a target object seen for the first time. To the best of our knowledge, we have not found in the literature other works that achieve such precision with a controller learned from scratch. Thus, this work presents a new system for autonomous robotic manipulation with high processing speed and the ability to generalize to several different objects.


Introduction
Object grasping is a canonical research area in robotic manipulation. However, since it is regarded as an open problem of robotics [1], its relevance and importance extend to the present day, as can be noted in the lack of dexterity with which robots manipulate complex objects [2].
There are infinite candidate grasps that can be applied to an object. So, as identified by Bohg et al. [3], a good subset of grasp hypotheses for finding the ideal grasp configuration comprises the type of task that the robot will execute, the features of the target object, what type of prior knowledge about the object is available, what type of hand is used, and, finally, the grasp synthesis.
Grasp synthesis is the core of the robotic grasping problem, as it refers to the task of finding points in the object dates by some metric. This process generally assumes the existence of previous experience of grasping, provided by means of heuristics or learning [3]. That is, these methods require annotations of those candidates that are considered correct [5], to serve as a model for the algorithms used. Therefore, one must generate these annotations in some way, either by means of real robots [6,7], simulations [8] or direct annotations in images [9].
Among the works that use empirical methods, many deal with some kind of visual information in their algorithms. The main benefit of using images for grasp synthesis is its independence from 3D models of the target objects. In real scenarios, it is often difficult to obtain an accurate and complete 3D model of a first time seen object [10].
Du, Wang and Lian [11] assume that the vision-based robotic grasping system is composed of four main steps, namely, target object localization, object pose estimation, grasp detection (synthesis) and grasp planning. The first three steps can be performed simultaneously considering a system based on convolutional neural network that receives an image of the object as input and predicts a grasping rectangle as output.
Regarding the grasp planning phase, in which the manipulator finds the best path to the object, the robotic grasping system must overcome some challenges to be applicable in the real world. It should be able to adapt to changes in the workspace and consider dynamic objects, using visual feedback. Therefore, the robot must track the object so that it does not leave the camera's field of view so that localization, pose estimation and grasp synthesis can be performed in a reactive way.
Most of the approaches for the robotic grasping task perform one-shot grasp detection and cannot respond to changes in the environment. Thus, the insertion of visual feedback in grasp systems is desirable since it makes it robust to perceptual noise, object movement, and kinematic inaccuracies [12]. However, even with modern hardware, classical approaches require a deterrent amount of time for closed-loop applications, and only perform adjustments of force, without visual feedback [13].
Thus, some works started to include a stage of visual servoing to deal with possible disturbances during grasp execution [14]. However, this technique heavily relies on the extraction and tracking of features, often based on 3D models, camera parameters and other information that must be known a priori. Although some recent approaches aim to minimize the work of feature design, control modeling, and the need for a priori information, they fail to achieve all of these advantages simultaneously.
A grasping system that can combine the benefits of automatic grasp detection and that receives visual feedback to deal with unknown dynamic objects, in real-time, is one of the goals in robotics. This work aims to advance towards this goal by designing a real-time, real-world, and reactive autonomous robotic manipulation system.
The main contributions of this work are: • Comprehensive review of works that use the Cornell Grasping Dataset for static grasping; works that introduce learning algorithms for visual servoing; and works that address dynamic and reactive grasping. • Design of a convolutional network that achieves stateof-the-art prediction speed in the Cornell Dataset and implementation of the network in a real robot, as an extension of a published conference paper [2]. • Design of four models of convolutional neural networks that can execute visual servoing in different target objects, by generating a proportional velocity signal based only on the reference image and the current image. The simplest model achieves stateof-the-art positioning error considering a controller learned from scratch dealing with a first-seen object. • Construction of a large dataset for end-to-end visual servo control using a Kinova Gen3 robot. • Implementation of all algorithms in a final grasping system able to consider distinct and dynamic target objects subject to different lighting changes. • Detailed analysis of processing speed, efficiency, and comparison with related works for all developed algorithms, both offline and in the Gen3 robot. This paper is organized as follows: Section II presents a brief survey of grasp detection, visual servoing, and dynamic grasping. Section III describes the methodology of the proposed system. Section IV presents the achieved results and discussion. Finally, Section V presents some conclusions.

Brief Survey
In this work, target object localization and object pose estimation are sub-tasks of the grasp detection stage and are processed through deep learning, in an approach called end-to-end grasp detection [11]. Object localization and object pose estimation are problems with their own literature and will not be addressed here, so greater emphasis is placed on grasp detection strategies.
The grasp planning stage is performed in two steps. First as a visual servo controller, to reactively adapt to changes in the object pose. And then, as an internal problem of the robot's inverse kinematics, with no restrictions on the movement of the robot to the object, except those related to singularities.
Thus, a brief review of the works on visual servo control is presented, with greater emphasis on the ones that use learning to synthesize the controller. Finally, works that use visual sevoing specifically in the task of grasping, here called dynamic grasping, are presented.

Grasp Detection
The early methods of grasp detection, known as analytic methods, rely on the geometric structure of the object to be grasped and have many problems with execution time and force estimation. Besides, they differ in many ways from data-driven methods. For a complete review of these methods, one can read the work of Bicchi and Kumar [15].
In the context of data-driven approaches, using only images, Jiang, Moseson and Saxena [16] proposed a representation of the position and orientation of a robot's grippers, just before closing for grasping, from five dimensions. The scheme is illustrated in Fig. 1. In the image, (x, y) is the center of the oriented rectangle, w represents the opening of the grippers and h its size, that is, the edges shown in blue represent the grippers of the robot. Finally, there is the angle θ that represents the grippers' orientation.
This five-dimensional representation is enough to encode the seven-dimensional representation of the grasp pose [16] since a normal approximation to the image plane is assumed, so that the 3D orientation is given only by θ. The images required for the algorithm design were obtained through the Cornell Grasping Dataset (CGD) [16]. Such dataset is composed of several images of domestic objects and their respective ground truth grasp rectangles.
Lenz, Lee and Saxena [9] were one of the first to test their algorithms in this dataset and to use deep learning for grasp detection. Although they did not obtain a relevant accuracy, it encouraged other researchers to investigate grasp problem using the rectangle representation.
Redmon and Angelova [17], for example, also used the CGD to train an AlexNet architecture [18] that receives RGD images as input and retrieves grasp rectangles. Furthermore, they introduced two different learning methods to accomplish grasp detection which are called direct regression and multigrasp detection. The former is based on the presence of a single observed object and the latter consists of estimating one rectangle for each fraction of the input image. Thus, using the multigrasp detection method, the AlexNet model is able to predict more rectangles associated with multiple objects.
Kumra and Kanan [1] developed two ResNet [19] based models that work in parallel. While one of them is trained with RGB inputs, the other one receives depth data from the CGD. The accuracy of their results surpassed the ones obtained by Redmon and Angelova [17] and, although the joint networks have a higher number of parameters, the authors report a prediction time of 103 ms.
Based on the Faster R-CNN [20], Chu, Xu and Vela [21] proposed a two-stage framework to address the grasp task. In the first stage, the grasp points are detected and, in the second stage the orientation of the rectangle is obtained by classification methods. This network, from the object detection point of view, is a two-stage detector whereas single-stage detectors perform direct regression of the rectangle. So, although the authors achieved good detection results in CGD, the grasp detection runs at only 8 fps.
Inspired by the same object detector network, Zhou et al. [22] proposed a new approach that regards the variation of the rectangle angles, instead of its scale and size. According to the authors, the definition of the grasp rectangle heavily relies on its angles, which justifies their choice. They achieved state-of-art results in terms of grasp detection accuracy. In addition, the built network is singlestage, but the prediction speed is increased by only 0.2 fps compared to Chu, Xu and Vela [21].
Morrison, Corke and Leitner [23] introduced a generative neural network (GG-CNN) that detects grasps, and an associated quality index, in all pixels of the input image, so, a fully convolutional architecture can be used. The authors used the CGD to transform the rectangular representation into parameterized images of Grasp Quality, Angle and Width, and use them for training. For testing, the network is evaluated in a real scenario with a prediction speed of 52,6 fps. However, the authors did not present the network efficiency in the CGD.
Following the same approach, Gu, Su and Bi [24] reparameterized the instances from the CGD into three images representing the ground truth grasp and use a fully convolutional architecture (AGN) as well. They introduced a mechanism of attention in the network so that the model focuses on salient features and, thus, reaches state-of-theart results in the dataset. However, the authors did not make it clear which set of the dataset is used to obtain this result. Furthermore, the developed network is slightly slower than GG-CNN and no real tests are presented.
Our work focuses on the development of a simple and efficient CNN that predicts grasp rectangles. In the training and testing steps, the proposed network is light enough to allow the joint application of a second CNN that addresses the task of visual servo control. Thus, the entire system can be executed in real-time for robotic applications without decreasing the accuracy for both tasks.

Visual Servo Control
Classic Visual Servoing (VS) strategies demand the extraction of visual features as an input to the control law. These features must be correctly selected since the robustness of the control is directly related to this selection. However, the use of projected features requires manual engineering for each new system and it avoids the exploitation of statistical regularities that make the developed VS system more flexible [25]. Assuming that notable convergence and stability issues can appear if the VS is performed in a region away from the projected operating point [26], the classic VS techniques mainly deal with controller convergence and stability analysis.
Relying on projected features, some studies sought to investigate the applicability of machine learning algorithms in VS [27]. Other authors also sought to automate the interaction matrix estimation using a multilayer perceptron [28,29] or support vector machines [30].
The first step towards the independence of feature extraction and tracking was taken with the development of Direct Visual Servoing (DVS) by Deguchi [31]. The technique has been improved so that it can better deal with nonlinearities present in the minimized cost function and convergence domain [32,33,34]. However, although the control law is obtained directly from pixel intensity, information about the objects is still necessary and the complexity of the image processing algorithms tends to compromise their use.
In order to create a control system that uses neither projected features nor metric information about objects, and that is still robust to imaging conditions, Silveira and Malis [33] developed a new DVS model. The authors explored the projective and geometric parameters that relate the current and desired image using the pixel intensity, however, the system had initialization issues since obtaining these parameters is computationally expensive. In a later work [34], the authors presented three optimization methods to try to overcome computational complexity. However, cheaper methods do not had good convergence properties and generally they need prior knowledge of the system. Furthermore, when evaluated in the real world, the algorithms converge only with reduced gains and the cheaper method did not converge.
In this sense, the most recent VS techniques have explored deep learning algorithms to simultaneously overcome the problems of feature extraction and tracking, generalization, prior knowledge of the system, and, in some cases, processing time. The first work that demonstrated the possibility of generating a controller from raw-pixel images, without any prior knowledge of configuration, was developed by Zhang et al. [35]. The authors used a Deep Q-Network to perform a target reaching task controlling 3 joints of a robot by means of deep visuomotor policies. The training was done in simulation and failed to be confronted with real images. However, when the camera images were replaced by synthetic images, the robot was controlled.
Other works that follow the reinforcement learning approach use deterministic policy gradients to design a new image based VS [36] or Fuzzy Q-Learning, relying on feature extraction [37], to control multirotor aerial robots. In a different approach, some works that explore deep learning in visual servoing do so through convolutional neural networks. Although Reinforcement Learning (RL) appears to be the best approach to the problem, by mapping joint angles and camera images directly to the joint torques, CNNs can also be trained to perform end-to-end visual servoing. Moreover, the generalization power achieved by CNNs is superior to that of RL, since the parameters learned by RL are specific to the environment and task [38].
Saxena et al. [38] developed a CNN trained to perform visual servoing in different environments, without knowledge of the scene's geometry. To this end, the authors trained a network, based on the FlowNet architecture [39], with the 7-Scenes dataset [40], which has several images taken in sequence through transformations in the camera. Thus, having the current image I c and the desired image I d , the network can predict the homogeneous transformation that relates the camera pose in I d and the camera pose in I c . The authors performed tests on a quadcopter and achieved good results in both indoor and outdoor environments.
Bateux et al. [41], like the previous authors, developed a CNN capable of predicting the transformation that occurred in a camera through two images. Some essential differences are in the architecture used, in the robot operated and, mainly, in the dataset used. The networks are based on fine-tunning of the AlexNet and VGG [42] networks, and the operated robot is a 6 DoF manipulator. The authors developed their own dataset using only one image. Starting from virtual cameras, it was possible to generate thousands of images and their associated transformations based on homography.
In our work, four models of convolutional neural networks were designed as potential candidates for end-to-end visual servo controllers. The networks do not use any type of additional information other than a reference image and the current image, to regress the control signal. Thus, the proposed network works as a de facto controller, predicting a velocity signal, not a relative pose.

Dynamic Grasping
Learning visual representations for perception-action [43] that adhere to reactive paradigm and that generates a control signal directly from sensory input, without highlevel reasoning [13], can help in dynamic grasping. Therefore, some authors developed variants of VS specifically for reaching and grasping tasks by using an uncalibrated camera [44], calculating the visual-motor Jacobian without knowledge of the system [45], or using policy iteration and imitation learning techniques [46,47] One of these works, developed by Lampe and Riedmiller [13], uses reinforcement learning to determine the outcome of a trajectory and to know if the robot will reach a final position in which closing the grippers results in a successful grasp. In addition, the authors proposed a grasp success prediction that anticipates grasp success based on current visual information, to avoid multiple attempts of grasping.
However, since they are applied to specific types of objects and since they still rely on some kind of prior knowledge, these systems do not have full capacity for generalization. Thus, more recently, a significant number of works have explored the use of deep learning as an approach to the problem of closed-loop grasping.
In this context, Levine et al. [48] proposed a grasp system based on two components. The first part is a prediction CNN that receives an image and a motion command as input and outputs the probability that, by executing such a command, the resulting grasp will be satisfactory. The second component is the visual servoing function. This uses the prediction CNN to choose the command that will continuously control the robot towards a successful grasp. Separating the hand-eye coordination system between components, it was possible to train the grasp CNN using standard supervised learning and design the control mechanism to use the network prediction so that grasp performance is optimized. The resulting method can be interpreted as a form of deep reinforcement learning [48].
Aiming to improve [48], Viereck et al. [12] developed a system based on CNN to learn visuomotor skills using depth images in simulation. The learned controller is a function that calculates the distance-to-nearest-grasp so that it can react to changes in the position and orientation of objects. Starting from simulation, and mounting the depth sensor near the robot end-effector, the authors managed to adapt the algorithm to the real world, avoiding the two months of training experience performed by Levine et al. [48]. However, the developed CNN calculates the distance in relation to a grasp pose given a priori. This grasp pose is obtained using the algorithm developed by Pas and Platt [49], so that errors in the grasp detection can induce errors in the control.
Inspired by the work of Viereck et al. [12] and Levine et al. [6], whose network receives actions as inputs, Wang et al. [50] developed a Grasp Quality CNN that relates these actions to the images that the robot would see when executing them, using homography. Thus, from a dataset of only 30K grasp trials, collected in a self-supervised manner, the authors removed the need to generalize over different actions and predict the grasp success metrics in a large number of poses.
In a way closer to the methodology developed in our work, Morrison, Corke and Leitner [51] developed a system for closed-loop grasping in which the grasp detection and visual servoing are not simultaneously learned. The authors used a Fully CNN to obtain the grasp points and apply a position-based visual servoing to make the pose of the grippers match the predicted grasp pose.

Methodology
To address the generalization challenge, a convolutional neural network is designed and trained for grasp detection on a large number of objects. To avoid one-shot grasp and consider changes in the workspace reactively, a visual controller is designed. Furthermore, to also satisfy the generalization condition in the controller, it is learned from scratch using a CNN capable of end-to-end visual servoing. The real-time execution of the algorithms is guaranteed, since the developed networks are, by design, both light and fast.
The purpose of the VS is to guide the manipulator, through comparisons between the image continuously obtained by the camera and a reference image, to a position where the robot has a full view of the object, so that the grasp detection conditions are met. Thus, the application of the method encompasses all situations in which a robotic manipulator, with a camera mounted in eye-in-hand mode, must track and grasp an object.
The system consists of three phases: design stage, test stage, and operational stage. The first is based on designing the CNN's architectures and training them, as well as gathering and processing the datasets. In the second phase, offline results are obtained using validation sets and evaluated based on their accuracy, speed and application domain. The third phase involves the implementation of the trained networks on a robot to assess their adequacy in real-time and real-world applications.
The requirement for the system operation in the operational phase is to obtain an image of the target object a priori, which will be used as a setpoint by the VS. The control loop is executed as long as the L1-norm of the control signal is greater than a certain threshold. The operational phase is outlined in Fig. 2.
A single reference image is presented to the system as one of Visual Servoing CNN's inputs. The image obtained by the camera at present time serves as the second input in this network and as the input for the grasp CNN. Both networks run continuously, as the grasp CNN predicts the rectangle in real-time for monitoring purposes and the VS network executes real-time control of the robot's pose.
The VS CNN predicts a velocity signal, which must be multiplied by a proportional gain λ, to be applied in the camera. The robot's internal controller performs all necessary calculations to find the joints velocities that guarantee the predicted velocity in the camera. At each loop execution, the current image is updated according to the robot's current position and, as long as the control signal is not small enough to guarantee convergence, this loop is repeated.
When the stop condition is satisfied, the prediction of the grasp network is mapped to the world coordinates. Then, the robot executes the inverse kinematics to reach the predicted points and close the grippers. Since no depth information is considered and mapping to world coordinates does not allow it to be found, the robot's approach to the object happens up to a height of Z = 23cm, by default.
The methodology used in all phases of the proposed system is presented in the following sections. All network illustrations were designed using the open software Plot-NeuralNet [52].

Grasping Network Architecture
The convolutional network architecture illustrated in Fig. 3 is proposed for grasp detection. The network receives as input a 224×224×3 image, in which the third dimension refers to the number of channels, i.e. RGB, without any depth information. The first layer is composed of 32 convolution filters with dimensions 3×3 and the second layer contains 164 of these filters. In both, the convolution operations are performed with stride 2 and zero-padding, and are followed by batch normalization and 2 × 2 max pooling.
The third layer contains 96 convolution filters in which the convolutions are performed with stride 1 and zeropadding, and followed only by batch normalization. The fourth, and last, convolutional layer is composed of 128 filters performed with stride 1 and followed by 2 × 2 max pooling.
After the last layer of convolutions, the resulting feature map is flattened in an one-dimensional vector with 4,608 elements. This vector is further delivered to two Fully Connected (FC) layers with 256 neurons each. Between these layers, a dropout rate of 50% is considered during training.
Finally, the output layer consists of 5 neurons corresponding to the (x, y, w, h, θ) values that encode the grasp rectangle. In all layers, the activation function used is ReLU, except in the output layer, in which the linear function is used. The implemented network is noticeably smaller than others described in the literature for solving the grasping problem, with 1,548,569 trainable parameters.

Cornell Grasping Dataset
To proceed with the grasp network training, the Cornell Grasping dataset is used [16]. It consists of a set of 885 images and associated point clouds of 240 common household objects, which would potentially be found by a personal robot. These objects have the appropriate size and shape to be grasped by a robotic arm equipped with parallel grippers capable of opening up to 4 inches. These dataset features make it possible to apply the trained agent not only in domestic environments, as seen in the related works.
To obtain the data, a Kinect sensor mounted in the robot's end-effector was used. During the data acquisition, the positions of the robot were chosen so that the manipulator can grasp the object from a path normal to the image plane. For encoding the dataset ground truths, the grasp rectangles were compiled with the x and y coordinates of the four vertices.
Based on the vertices, five parameters are obtained for their representation, according to the methodology found in Jiang, Moseson and Saxena [16]. Fig. 4 and the equations 1, 2 and 3 show how the dataset is reparameterized into (x, y, w, h, θ).
The x c and y c parameters, which represents the x and y coordinates of the rectangle's center point, respectively, can be obtained from: The grippers opening, w, and height, h, are calculated from the vertices as follows: Finally, θ, which represents the orientation of the grippers relative to the horizontal axis, is given by:

Data augmentation
Data Augmentation (DA) [53] is a strategy for artificially increasing training data to improve network's learning ability. This technique is especially important if the relationship between the amount of available data and the amount of trainable parameters in the network is too small. The applied method differs from the traditional translations and rotations usually applied in DA. Assuming that a much larger number of transformed images, each one with a considerable amount of visual cues, can aid the network's learning process, we use a 320 × 320 cropping window that slides in the image, ensuring that the object of interest is always in the cropped area. Fig. 5 illustrates the strategy. In order to crop the image without part of the object being outside the representation, the values of the labeled grasp rectangles were taken into account. Thus, the x and y displacements of the window were limited by the minimum x and y coordinates of the associated grasp rectangles, respectively. Based on the position of the crop window, appropriate adaptations on the original ground truth rectangles was also made, to serve as label for the new dataset instance.
The applied DA resulted in a new dataset more than 335 times larger than the original: 297,338 training instances obtained from just 885 images.

Training and Evaluation
The network is implemented in Python through the Keras framework (with Tensorflow backend), in the Ubuntu 16.04 operating system. The hardware used is an Intel Core i7-7700 processor with 8 cores of 3.6GHz and an Nvidia GPU GeForce GTX 1050 Ti. We used mean squared error as loss function, Nadam [54] with a learning rate of 0.002 as optimization algorithm, batch size of 144 and 30 epochs of training.
The dataset is divided into training, validation, and test sets, and the number of samples in each of them is presented in Table 1. Another division performed in the dataset comprises the type of images contained in the above mentioned sets. In the case called Image-Wise (IW ) the figures are randomly organized into sets. In the case Object-Wise (OW ), all images containing the same object are placed in the same set. The evaluation of the result is made with the follow-ing metric, as used in related works of grasp detection by means of the rectangle representation: • the difference between the angle θ of the predicted rectangle (r p ) and the correct rectangle (r c ) must be within 30 degrees; • the Jaccard index of the predicted rectangle relative to the correct one must be greater than 0.25. This index represents the percentage of overlap between the rectangles, according to the following formula: If the network prediction satisfies these two conditions for one or more of the labeled rectangles associated with the object, then this prediction is considered to be correct. Note that the overlap between label and prediction (0.25) is less rigorous than in other computer vision problems (0.5) since the labeled rectangles represent only a fraction of all possible grasping rectangles associated with an object. Furthermore, Redmon and Angelova [17] claim that the limit of 0.25 is often sufficient to enable a successful grasp.

Visual Servoing Network Architectures
Unlike grasping, the network designed to perform visual servo control of a robotic manipulator receives two images as input and must regress six values, which can also be categorized into two outputs, considering linear and angular camera velocities. To this end, 3 network architectures are designed. Because two models were created from one of these architectures, a total of 4 models are presented to approach the VS task.
All networks, however, are generated from the remarkably simple structure of the grasping network. Therefore, the number of filters and the operations considered in each convolutional layer, the number of FC layers and the respective number of neurons, as well as the dropout rate are similar to the grasping network and common to all developed models. All networks receive two image inputs in the format 640 × 360 × 3 and output the 6-dimensional velocity vector without any intermediate step.
The first model for the VS task, called Model 1 -Direct Regression, is presented in Fig. 6. It is basically identical to the grasping network, except for the inclusion of maxpooling in the third convolutional layer and for the different input dimensions, which causes the same proportional difference on the feature maps. The inputs are simply concatenated and the data flow inside the network is always ahead, without any association or division of feature maps. Model 1 has only 1,549,690 trainable parameters.
The second model, called Model 2 -Task-specific Regression, is presented in Fig. 7. The network inputs are concatenated and the third set of feature maps is processed by two separate layers sequences. Thus, the 6D velocity vector is predicted by the network in the form of two 3D vectors. Specifically, this structure consists of a shared encoder and two specific decoders -one for linear velocity and another for angular velocity. Model 2 has 2,906,106 trainable parameters.
Therefore, the particularities of each domain can be learned by the network at the same time that its similarities can help in generalization. This structure is known as a multi-task network and, as presented by Kendall et al. [55], the cues from one task can be used to regularize and improve generalization of another one, by using inductive knowledge transfer. The intention is to evaluate how the dissociation of the velocity control problem in two specific tasks of linear and angular velocities control, can affect the network accuracy.
The third and fourth models, called Model 3 -Direct Regression from Concatenated Features and Model 4 -Direct Regression from Correlated Features, are created using the architecture presented in Fig. 8. This structure has two encoders and one decoder, which makes it possible to obtain high-level representations of each image before associating them. Two different association operators (Σ) were considered: a simple concatenation, which defines Model 3, and a correlation layer, used in the work of Dosovitskiy et al. [39], which defines Model 4.
Model 3 simply considers the concatenation of the feature maps resulting from the third convolutional layers, so that the input to the fourth layer is twice as deep. Model 4, on the other hand, has a correlation layer that aids the network to find correspondences between the feature representations of each image. The original correlation layer is a structural unit of the optical flow network FlowNet [39]. Models 3 and 4 have, respectively, 1,850,286 and 1,533,358 trainable parameters.

Visual Servoing Dataset
To train a model that can perform visual servoing on different target objects, without the need to design features, it is necessary to have a dataset that efficiently captures the attributes of the environment in which the robot operates, be representative of the VS task and diverse enough to ensure generalization. To this end, the data is collected by the Kinova Gen3 robot, in a way that approximates the self-supervised approach. Human interventions are associated with the assembly of the workspace and the setup of the robot, which involves determining a reference pose from which the robot captures the images and labels them.
The robot is programmed to assume different poses from a Gaussian distribution centered in the reference pose, with different standard deviations. This approach is inspired by the work of Bateux et al. [41], which does the same but using virtual cameras and homography instead of a real environment. The reference pose (mean of the distribution) and the sets of Standard Deviations (SD) assumed by the robot are presented in Table 2.
The SD choices take into account the expected displacement values that the robot must perform during the VS. Images obtained from a high SD help the network to    Fig. 9.
Regarding the choice of objects and their distribution in the workspace to build the dataset, two factors were considered. On the one hand, the network must learn the features that make it possible to correlate a reference image to the current one, therefore, the greater the number of visual cues in the dataset images, the better the network's learning ability. Thus, some scenes comprise a large number of objects, with varied structures, scattered across the table (as in Fig. 9). On the other hand, some scenes are constructed so that the network learns the task in which it will be applied, that is, grasping. Therefore, only one object is placed on the table, so that little visual information is available, but a good representation of the task is obtained.
In addition to single and multiple objects, two other types of scenes were considered. A framed map with a reflective surface, so that the network is robust to distortions in the captured image, and sets of books so that the network learns relevant features in flat objects. Brightness changes were considered during the dataset generation, since it was captured at different times of the day, without the concern of keeping luminosity constant. In addition, shadows in random regions of the image were considered, since the environment was not controlled and susceptible to intermittent transit of people. The only post-processing performed on the obtained images is the exclusion of those that do not contain any part of the object. Table 3 shows the dataset composition in detail. After obtaining the data, the dataset is structured in the form (I, [x, y, z, α, β, γ]), in which I is the image, and [x, y, z, α, β, γ] is the associated camera pose when this image was captured, with α, β, γ represented in the Tait-Bryan extrinsic convention. In order to use this dataset to train a Position-Based VS neural network -according to the Visual Servo Control Tutorial by Chaumette and Hutchinson [56], two images and the relative pose between them must be considered. Then, each instance of the processed dataset takes the form (I d , I c , d H c ), in which I d is a random instance chosen as the desired image, I c is another instance chosen as current image, and d H c is the transformation that relates the current frame to the desired camera frame. This is done by expressing each pose, represented by translations and Euler angles, in an homogeneous transformation matrix form ( 0 H d and 0 H c ), and then obtaining d H c = 0 H −1 d 0 H c . Finally, for the network to be, in fact, a controller, the intention is that its prediction is directly the velocity signal of the camera, i. e., the control signal. Hence, the data structured as (I d , I c , d H c ) is transformed to (I d , I c , v c ), in which v c is the proportional camera velocity. The propor-   tional term is used because the λ gain is not considered in determining the labeled velocity, and must be tunned a posteriori, during the control execution. The velocity v c is obtained from d H c using: where R is the rotation matrix, r ij is the element at line i and column j of the same matrix, c * t c is the translation vector of the current camera frame related to the frame of the desired pose, λ is the proportional gain (set to 1 in the dataset and later adjusted during the operational stage) and θu is the angle-axis representation of the rotation matrix.

Training and Evaluation
The networks are implemented in Python using Keras (with Tensorflow backend), in the Ubuntu 16.04 operating system. The hardware used is an Intel Core i7-7700 processor with 8 cores of 3.6GHz and an Nvidia GPU GeForce GTX 1050 Ti. We used mean squared error as cost function, Adam [57] with learning rate of 0.001 as optimization algorithm, batch size of 32 and 10 epochs of training.
For evaluation, the mean squared error is used as the performance metric, according to Eq. 7, where n is the number of instances in the validation set, v l c is the labeled camera velocity and v p c is the network prediction.
For the purpose of illustration, predictions of all models for the same validation instance are implemented as a velocity command in the robot. No feedback is considered, since the intention is only to show the effect of applying the velocity predicted by the network in open-loop, so that the mean squared error of this prediction is better understood in terms of the difference between poses.

Results and Discussion
This section presents all the results obtained with the developed algorithms and analyzes them in two different ways; (a) considering the tasks of grasp detection and visual servoing in isolation and (b) the joint application in the dynamic grasping system.

Offline Results
This section presents the results achieved for grasping and visual servoing from the dataset validation sets developed for each task.

Grasp Detection
The visual results of the developed grasp detection algorithm are illustrated in Fig. 10. It is possible to note the network's ability to predict a grasp rectangle consistent with the object's shape even if it requires a small gripper opening or a choice from several options of grasp points.
Since several labeled rectangles are associated with each dataset instance, the way training is conducted must be chosen between using all of them as labels or using only one, chosen at random. Although the first option allows a more reliable characterization of the grasping process, it is not exactly suitable in the case where the network predicts only one rectangle, as this prediction ends up tending to the average of the labeled rectangles. Thus, in the case of a circular object, for example, the network processes all labels arranged on its circumference and returns a central rectangle, which will certainly not result in a good grasp. Hence, we carry out all training steps for the grasp task with only one label associated with each object.  Table 4 presents the speed and accuracy obtained by our work and compares it with related works. Our accuracy of 94.8% in the image-wise division is in the top three, and the 86.9% accuracy for the object-wise division is comparable to the listed publications, except for the works of Chu, Xu and Vela [21] and Zhou et al. [22], which are far behind in speed. Considering the extremely lightweight architecture of the projected network (4 convolutional layers and 2 FC layers), it is fair to say that it is quite efficient. With a much lower computational cost, expressed in high prediction speed, our proposed network can achieve similar or better results than state-of-art methods which employ more complex architectures. Moreover, many of the listed works also use depth information in addition to RGB images or replacing one of the color channels, making their application inflexible to scenarios where such information cannot be obtained.
Our network's prediction time surpasses the one reported in the work of Park and Chun [63] by approximately 10ms, updating the state-of-the-art prediction time on the Cornell Grasping Datatset from 23ms to 13.5ms. Considering that many of the works in Table 4 use more powerful GPUs than ours, the prediction speed comes primarily from the simplicity of the developed architecture. In addition, the small penalty on accuracy is achieved with the robust data augmentation strategy employed. This high prediction speed allows us to use the network in real-time robotic systems in order to make them able to react to changes in the environment and in the position of the target objects.
For a final comparison in terms of accuracy and prediction speed using the same hardware, we replaced the covolutional layers of our network by a ResNet-50 v2 [19]. With the ResNet as encoder, our framework achieved accuracy of 95.44% in the image-wise division but was unable to generalize learning to new objects in the object-wise division, reaching just 46.92% of accuracy. ResNet's large number of trainable parameters (25 million) specialized to training data and lost the ability to generalize, resulting in overfitting.
Regarding the training time and prediction speed, the framework using ResNet as encoder took twice as long as our network to be trained and presented a prediction speed approximately three times lower than ours (25.64fps).

Visual Servoing
Prediction speeds and accuracy for the developed networks are presented in Table 5. The speeds shown in the table were obtained from the average of the prediction speed of each model for the validation data. The Mean Squared Error (MSE) is the evaluation metric. Model 2 is the one that leads to the best offline results in the visual servoing task, demonstrating that the dissociation between linear and angular velocities and multi-task learning can boost performance. Model 4, on the other hand, had the worst performance, which suggests that a correlation layer designed for the optical flow task does not produce significant features for the visual servoing task.
By offline results, we mean only the difference between the predicted and labeled velocities. To get a better idea of how the MSE expresses the quality of the designed networks, the prediction of each model for an instance of the validation set is presented in Table 6. Thus, it is possible to make a quantitative comparison between the predicted and the labeled velocity values. Going further, it is possible to use these predictions to illustrate the robot's behavior when these speeds are applied to the camera. First, the robot is positioned in the pose equivalent to the one it had when it captured the current image (I c ) of the validation instance. Then the speed command is applied (without feedback), considering λ lin = 0.1 and λ ang = 10, until the L1-norms of the linear and angular velocities are greater than the L1-norms of the previous iteration. When this condition is met, the applied speed is set to zero and the robot stops moving. This is not the actual visual servoing, which is performed in closed-loop, but rather an illustration of how the velocity predicted by the network can affect the robot's pose.
The pose of the robot is read throughout the process and graphically illustrated in Figs. 11 and 12. The best scenario is represented by the application of the Model 2 prediction (Figs. 11b and 12b).
It must be said that these results may be different according to the chosen λ, which interferes both in the con-vergence time and in the final positioning of the robot. However, as the same experimental conditions were kept for all models, it is possible to make a comparative analysis of efficiency. The largest positioning error for Model 1 is in z, which differs by 21mm from the expected. The biggest error in Model 2 is also in z, of only 9mm. Model 3 has an error of 16mm in y. Model 4, although it has the largest MSE among the models, is the one that achieves the least positioning error, of 8mm, in z. This may indicate that, although in general the values predicted by Model 4 are more distant from the expected values, they are proportional, which plays a fundamental role in visual servoing.
Concerning orientation, the convergence of control occurs when the line changes its slope. Motion from that point on is just noise, which would be suppressed if there were feedback. The angular displacement that the robot must make in this validation instance is not so significant, but it still shows that the networks, especially Model 2, can achieve accuracy of less than 1 degree in all directions.
Figs. 11 and 12 clearly show how well adapted the networks are for the visual servoing task, as they show the difference between the final pose achieved by the robot and the desired one. However, the figures do not accurately express the robot's behavior when the networks are used as actual controllers, in a closed-loop way, with visual feedback. Furthermore, although they demonstrate generalization for tuples of images not seen in training, they do not demonstrate generalization for unseen objects. These two scenarios are explored in section 4.2.2.

Online results
This section presents the achieved results when the developed algorithms are implemented in the Kinova Gen3. Therefore, suitability for the real world and time constraints can be assessed.

Grasp Detection
The grasp detection results are visually presented, showing the predicted grasp rectangle in the image acquired by the robot for different objects. Once the rectangle is correctly predicted, the grasp can be executed considering a 2D → 3D mapping that uses the camera's intrinsic and extrinsic parameters and a fixed depth for the gripper. The network application scenario is quite different from that represented in the Cornell dataset, primarily due to the different objects, but also due to the considerable difference in luminosity. Fig. 13 shows eighteen detection examples.
In general, the figure makes it evident that the predicted rectangles (grippers indicated in blue) are consistent with the shape of the target objects and can adapt to their sizes and orientations. Some particular cases, however, deserve more attention.
The objects represented in Figs. 13a and 13e are the easiest for the network to detect grasps, as they are large (d) Model 4 prediction Figure 11: Evolution of the robot's position in time (black) and its desired value (blue), when the network prediction (weighted by λ lin = 0.1 and λang = 10) for some instance of the validation set is sent to the robot, which executes it without feedback.  in length and small in thickness. Most of the objects that are commonly handled follow this pattern and therefore have a large number of instances in the Cornell dataset. The chalkboard eraser (Fig. 13b) and pencil case (Fig.  13f) are not as simple as the previous objects because they are thicker. The network needs to reason whether the object is so large that it exceeds the represented gripper opening. This same situation occurs with circular objects, as illustrated by the detections in the hand massage ball (Fig. 13g) and anti-stress ball (Fig. 13h), in which the smaller ball is easier to detect. However, circular objects can also confuse the network due to the infinite possibility of orientations.
The masking tapes represented in Figs. 13c and 13i also exemplify the thickness problem. But the great challenge regarding these objects is with the ring format. Often the network may reason that the middle of the object is a constituent part of it, especially if the object and the background have similar colors. Besides, the prediction of a grasp rectangle in the middle of a circular object (because of the arrangement of its labels), is only surpassed because only one label is used during training.
Cups and mugs can also offer a challenge to the network, as they can be caught both by the edges or by the handles. In both Figs. 13d and 13m, the network predicts a rectangle in the handles, which, especially in the case of Fig. 13m, can be justified by the prominence of the handle and the fact that the edges are not so distinguishable.
The ruler (Fig. 13k) and hand sanitizer (Fig. 13q) are transparent and, especially in the case of the ruler, reflective, however, the network manages to distinguish them. The scissors in Figs. 13n and 13o and the device in Fig. 13r, test the network's ability to reason on scale. The first pair of scissors is further away from the camera, so the network prefers to predict a large rectangle, whereas the second pair is closer, and the network can have a better understanding of how big the hole between the handles is, so it can make a more skillful grasp. The device in 13r exemplifies the same situation, however, whether near or far, the network always prefers the larger opening, since it understands that the gripper does not fit in the hole.
To end the discussion on Fig. 13, the objects in Figs. 13j, 13l and 13p test the network's ability to find the best grasp location, amid a large number of possibilities. It is possible to note that the network makes wise choices, especially regarding the laptop charger, when choosing the part of the object that is most likely to result in a stable grasp.
A final analysis of the network involves its behavior when the input image has multiple objects, as presented in Fig. 14. The network is designed to deal with only one object, which can be seen as a limitation, or as a strategy to avoid post-processing of choice between the predicted rectangles. The important thing is to note that the presence of two or more objects does not confuse the network so that it makes the prediction somewhere between them, on the contrary, it chooses one of the objects as a target. Images 14a and 14b show that this choice is made at random. This attribute is important because, although the application space involves only one object, the network operates normally if another object enters the camera's field of view.

Visual Servoing
To test the networks in a real visual servoing scenario, and assess their robustness to lighting changes (global or local), their ability to generalize to first-seen objects and to ascertain whether the prediction speed ensures realtime operation, these trained networks were implemented to control the Kinova robot. The difference between achieved and desired poses using visual servoing with the developed CNN model are presented in Table 7, considering λ lin = 0.05 and λ ang = 2. The gains chosen are less than those used in the offline experiment, as the network must deal with a noisy scenario. In the final dynamic grasp experiment, the convergence time becomes a relevant factor and is better considered by increasing λ. In general, it is possible to note that all networks have more difficulty in inferring the necessary velocity in z, to reach the desired pose. This can be seen both in the final positioning error and in the behavior of the control signal z, more oscillatory than the others.
The efficiency of the networks must be analyzed assuming that the target object was not an instance of training (it was considered only among other objects, not as a single object). Thus, the final errors of positioning and orientation express the network's ability to generalize a task that, in the literature, is almost always performed starting from a strong modeling and a priori knowledge of the object and camera parameters. Then, a VS controller designed using prior knowledge will certainly outperform one learned from scratch [13]. The intention is to show that this VS CNN can be applied in the field of autonomous manipulation in situations in which knowledge about the process is scarce or difficult to generate. However, designed networks are still able to obtain final millimeter positioning errors and less than one degree orientation errors. Especially Model 1 (Fig. 16), which is simpler and with higher prediction speed, surprisingly manages to obtain a final positioning error of less than 5mm. All other networks can achieve this same performance, but only in one direction, while others reach the order of centimeters.
Regarding the trajectory made by the robot during the control, again, Model 1 is the one that has the most desired behavior (Fig. 15). This is the closest to a pure straight line in 3D space, something obtained in classic PBVS, using s = ( c * t c , θu), only when the parameters involved in the control law are perfectly estimated. Figure 17 presents the trajectory, position and orientation in time for all other models. Model 2, which obtained the lowest MSE in the offline phase, ends the control with a relatively high error in positioning, however, it is the one that converges better in z. Regarding the orientation, it can be noted that this is the model that leads to the faster stabilization of the desired angles, something that can be attributed to the dissociation of the network outputs between linear and angular control signals. behavior of the control signal, presented in Fig. 18, generated by Model 4 is the closest to a default controller, in which speeds tend to zero quickly and no longer oscillate after that. Model 3, on the other hand, generates an extremely noisy control signal, which penalizes the trajectory made by the robot.

Dynamic Grasping
The last experiment carried out on the robot is the test of the entire system, as illustrated in Fig. 2. To do this, the robot takes a reference image of an anti-stress ball and then performs a displacement [∆x, ∆y, ∆z, ∆α, ∆β, ∆γ] = [0.15m, -0.15m, 0.05m, 5 • , -5 • , 5 • ] , from where control starts. To mimic the behavior of an object that changes its desired position after the grasping attempt is started, it was preferred to change the pose of the robot, which has the same effect and allows for better monitoring and reproducibility.
The specific ball was not used in the grasping dataset, or in the control dataset. Therefore, the experiment evaluates the adaptation of the robot to a dynamic first-seen object using a method entirely based on learning. Only Model 1 is considered since this is the one that achieves the best results in the robot. Fig. 19 illustrates the environment in which the robot is assembled and shows each step of the process. In Fig.  19a the robot setup is shown, highlighting the initial pose, followed by the pose in which the robot captures the reference image and the final pose, from where the dynamic grasping starts.
In Fig. 19b, three images of the robot are shown at different moments of the visual servoing, with the last image illustrating the robot's pose in the instant before the grasp execution when the stop condition is reached. Below each image of the robot, the differences between the current images of each instant and the desired image are also illustrated. The desired position of the ball is represented in blue and the current position in green. During the entire process, it is possible to see the prediction of the grasp network, so that the user can intervene at any time s/he thinks that the grasping can be performed, according to the prediction seen.
When the stop condition is reached, in which the current position of the ball is close enough to the desired position, the robot starts to approach the object for grasping. Fig. 20 shows some moments of the grasp execution. In the first and second images, the robot adjusts the orientation and opening of the grippers and also moves towards the (x, y) position after mapping the network prediction. Depth information is not used, so the final z value is fixed. In the third image, the robot closes its grippers until the applied force reaches a threshold, and then takes the grasped ball to a pre-defined position.
The behavior of the robot and the applied control signal are illustrated in Fig. 21. The robot comes close to convergence only in y, but as seen by the difference between the current and desired images in Fig. 19, it was enough to considerably reduce the distance to the desired position. The behavior in z starts in the direction of divergence, but after iteration 60, it starts to converge. The angular velocity signal was still relatively large, in magnitude, when the VS was interrupted.
Therefore, the choice of the considered L1-norm threshold must be the trade-off between the positioning accuracy and the speed with which grasping is performed. In a dynamic grasping scenario, speed naturally plays a more important role, so the system should perform the grasping as soon as possible, whereas the visual servoing step serves much more the task of keeping the object in the camera's field of view, than positioning accuracy.
For this reason, a relatively high threshold was chosen, so that the execution of the control does not take too long. It took 14s for the robot to stop visual servoing and perform grasping. This time can be reduced by considering higher gains and/or thresholds.
A second experiment to analyze the robot's behavior concerning a dynamic object is performed, in which a quick and unexpected change in the object's position is considered, as well as the case where the object moves with ir-  regular speed. During the experiment, the position of the object is not annotated, so the description of the motion performed by the object is purely approximate and visual, illustrated by Fig. 22. Robot's trajectory is illustrated in Fig. 23. The object considered for tracking is a screwdriver (not seen in training), whose desired position is represented in blue in the image. In the first sequence, the same conditions as in the previous experiment are maintained, that is, the robot takes an image in a certain pose, which serves as the desired image, and then moves to another pose to start the control. Sequence 2 occurs right after the object is abruptly moved to the right, causing the robot to track the object and then adjust the position.
During Sequence 3, the screwdriver is moved with approximately constant speed in y (robot base frame), causing the robot to follow the movement with a certain offset between the current and desired images. Finally, in Se-quence 4, the robot adjusts its position after the object stops moving, and, as soon as the stop condition is met, the control ends and the grasping is performed. Fig. 24 graphically represents all stages of the experiment and shows the 3D trajectory made by the robot, highlighting the transition between sequences. From the linear speed signal, it is possible to notice that the robot was about to stop the visual servoing and perform the grasping, but the sudden change in the position of the screwdriver causes an overshoot in the signal. The position graph shows that the correct y position is quickly reached and only maintained until the beginning of Sequence 3. From this moment on, the y position of the robot progressively increases as the object moves in that same direction.
From the beginning of Sequence 4, the y position is only adjusted to match the desired image, until the control ends. It is interesting to note, however, that z varies  greatly in response to these changes, even though it is not directly involved in them, while x remains almost constant, as expected. It may indicate that it is necessary to adjust a particular gain forż, or to retrain the network considering larger variations in z. At the end of the control, the speeds become very noisy because the floor appears in the image, confusing the network. However, even so, the stop condition is reached at a certain moment, and the grasping is successfully performed. A final experiment to evaluate the dynamic grasping system was performed in order to give an idea about the efficiency of the network for different objects and different initial poses. Far from being a statistical survey, but capable of validating the network's applicability, 20 grasping attempts in a dynamic scenario were considered for four objects.
The robot takes an image at a pre-defined position and then is repositioned considering a Gaussian distribution with standard deviations [σx, σy, σz, σα, σβ, σγ] = [50mm, 50mm, 50mm, 2 • , 2 • , 2 • ]. The results are shown in Table 8, indicating a success rate of 85%, in which, of the three failure cases, two occurred due to errors in the grasp network prediction, and one due to errors in the VS CNN.
Viereck et al. [12] reported a success rate of 77%(58/75) when considering dynamic objects in a reinforcement learning scenario. Morrison Figure 21: Results for VS in the dynamic grasping scenario: the robot is positioned [∆x, ∆y, ∆z, ∆α, ∆β, ∆γ] = [0.15m, −0.15m, 0.05m, 5 • , −5 • , 5 • ] from the desired pose, but as soon as the control signal's norm is below 0.05, the system perceives that it no longer needs to track the object, so the grasp can be executed other hand, develop a reactive grasping system quite similar to the one proposed, except for the fact that the visual servoing is a classic PBVS and dependent on grasping prediction, that achieves 81%(76/94) accuracy. Both works considered cluttered scenes, which may difficult grasp detection, but not the dynamics of the set, which, in all attempts, is only moved 10cm in one direction and rotated 25 degrees in z, which does not reproduce a challenging  [51] shows results. The authors reached 86% efficiency in 200 attempts, but, again, the control is done by a classic PBVS, depending on the pose predicted by the grasping network and the dynamics of the scene is the same as the one presented for the cluttered scenario. Thus, although the comparison is not necessarily fair, since the methods consider different objects and a much larger number of attempts, it can be admitted that the developed system has greater applicability to deal with the task for which it was designed, especially considering the fact that none of the target objects were in the training phase.

Conclusions
In this work, we addressed the problems of grasp detection and visual servoing using deep learning, and we also applied them together as an approach to the problem of grasping dynamic objects.
A convolutional neural network is employed to obtain the points where a robot with parallel grippers can grasp an object using only RGB information. In order to use a simple network with a small number of parameters, the data augmentation technique is employed innovatively, allowing to extract as much visual information as possible from the dataset images. As a result, the trained network can predict grasp rectangles at speeds that surpass the state-of-the-art in the Cornell Grasping dataset, without harshly penalizing accuracy. The trained network is also evaluated on a robot, with its own camera to test its generalization to different objects which are not seen in the training. The visual results demonstrate that the network is robust to real-world noise and is capable of detecting grasps on objects of different shapes, colors, in different orientations and subjected to considerable lighting changes.
Four models of convolutional neural networks were designed as potential candidates for end-to-end visual servo controllers. The networks do not use any type of additional information other than a reference image and the current image, to regress the control signal. These networks were tested both offline and on a robot to assess applicability in a real-world and real-time scenario. The simplest model was able to achieve a millimeter accuracy in the final position considering a target object seen for the first time.
To the best of our knowledge, we have not found in the literature other works that achieve such precision with a controller learned from scratch.
Finally, the trained networks that obtained the best results in each task, were embedded in a final system for grasping dynamic objects. In different scenarios, all a priori unknown to the network, the robot was able to keep the target object in the camera's field of view using the realtime visual servoing CNN. When the robot understands that it is close enough to the desired position, the prediction of the grasping network, which is also obtained in real-time, is mapped to world coordinates and the robot approaches the object to execute grasp.
The system at the current stage has the deficiency of not considering depth information during the execution of the grasping. In future work, the intention is to use information from an infrared sensor in the operational stage, maintaining training only with RGB, as this is seen as an advantage. Other considered adjustments relate to the determination of control gains, adaptation of the grasping network to predict multiple rectangles, obtaining grasp dataset in a self-supervised manner with the robot, and making the visual servoing dataset publicly available.
The trained algorithms were not designed to surpass those modeled from prior knowledge, but so that they can be applied in situations where the knowledge of the system is limited or impossible to be obtained. To this end, it is concluded that the developed methodology has superior applicability due to its high capacity for generalization, simplicity, speed, and accuracy.