RGB-D-Based Robotic Grasping in Fusion Application Environments

Abstract

Although deep neural network (DNN)-based robot grasping has come a long way, the uncertainty of predicted results has prevented DNN-based approaches from meeting the stringent requirements of some industrial scenarios. To prevent these uncertainties from affecting the behavior of the robot, we break down the whole process into instance segmentation, clustering and planar extraction, which means we add some traditional approaches between the output of the instance segmentation network and the final control decision. We have experimented with challenging environments, and the results show that our approach can cope well with the challenging environment and achieve more stable and superior results than end-to-end grasping networks.

1. Introduction

Robotic grasping is one of the most effective solutions for handling grasping work in challenging environments. In nuclear fusion applications, many tasks need to be carried out by the robotic arm, including robotic grasping, as shown in Figure 1.

As a fundamental robotic task, research into robotic grasping has been carried out for a long time. In the early works, researchers need to use the traditional Computer Vision (CV) algorithms to obtain the right grab point. Just as in [1], researchers use a single camera to observe the same towel from different angles of view, implement the border classification and corner grasp point detection algorithms based on these data, and eventually achieve the towel folding task. These algorithms were quickly replaced by Convolutional Neural Network (CNN)-based algorithms after the rise of deep learning. Thus, in this paper, we focus on learning-based approaches.

In [2], based on a monocular camera, Levine et al. proposed a learning-based approach to hand-eye coordination for robotic grasping, and they focused on training the CNN to understand the spatial relationship between the gripper and objects in the scene so the neural network (NN) could predict the probability that the task-space motion of the gripper will result in successful grasps.

Due to the nature of the robotic grasping task, the final target is to drive the gripper to the optimal gripping pose; the best input data are the 3D spatial coordinates that are hard to obtain from a monocular camera. Thus, after the RGB-D camera was introduced and matured, it quickly became a popular sensor for researchers. Now, the input data of most learning-based robotic grasping methods are based on the RGB-D messages provided by the depth cameras. Just as in [3,4], these approaches output the appropriate grasping rectangle for each object through the CNN structure. The centre of the rectangle is the grasp point. The long side and the short side of the rectangle correspond to the grasp angle and grasp width, respectively. Morrison et al. [5] introduced a Generative Residual Convolutional Neural Network (GR-ConvNet) to infer multiple grasp rectangles for multiple objects in the clutter in real-time; they also output a quality for each grasp pose. On the basis of [5], Kumra et al. [6] adjusted the structure of the neural network and achieved a better result on accuracy and efficiency.

In [7], researchers proposed a generative adversarial network based on monocular RGB images which can achieve an outstanding grasping result both in simulation and realistic environments. Tremblay et al. [8] proposed a 6-DoF pose estimation NN which could output the semantic label of each object. The whole training set for the NN is also built on synthetic data. In [9,10], the RGB-D information is converted into point clouds and neural networks are developed to give contact points for each object separately based on object recognition in the 3D space.

In [11], Wen et al. proposed a method that leverages hand-object contact heatmaps generated in a self-supervised manner in simulation to achieve a dense, point-wise grasping network, and they also introduced the “Non-Uniform Normalized Object Coordinate Space” (NUNOCS) representation for learning category-level object 6D poses and 3D scaling, which allows non-uniform scaling across three dimensions.

With the increasing number of approaches based on generative data, researchers are placing greater demands on the quality of the generated models. In order to make the virtual data as close as possible to the real model, Tobin et al. [12] explored a data generation pipeline for training a deep neural network to perform grasp planning that applies the idea of domain randomization to object synthesis. Meanwhile, Varley et al. [13] proposed a generation network based on CNN, which could complement the 3D shape of an object with an incomplete points cloud from several points of view. With adequate offline training, the generation network could complement the 3D shape accurately and rapidly.

Obtaining training data is always the most important and most difficult part of the supervised learning-based method. Some researchers are beginning to explore the use of unsupervised learning to solve the robotic grasping problem. In [14], researchers trained a vision-based closed-loop grasping reinforcement learning (RL) agent in a simulation environment, then transferred it to the real world and completed the grasping task successfully. Joshi et al. [15] presented a deep reinforcement learning-based method to solve the problem of robotic grasping using Visio-motor feedback. Quillen et al. [16] compared various RL algorithms in a virtual and realistic environment and selected an off-policy algorithm that could handle the task well.

From these related works, the learning-based ones exhibit several trends:

• Using the generated data as training data and translating the results from virtual to real;
• Transfer the task from individual objects to cluster objects grasping.

Most robotic grasping methods focus on building an end-to-end grasping network that could output a result that contains the grasp point, pose, and width. This is the right development direction and works well under ideal circumstances. However, in the fusion application environment, all components, including the interior of the vacuum chamber and the components to be assembled, are made of the same metal materials and have smooth surfaces. Almost all the RGB-D cameras’ performance on the non-optimal smooth surfaces with specular reflection is poor. The CNN-based grasping network algorithm can hardly yield stable results in such an environment.

Meanwhile, due to the black-box nature of the processes within neural networks, researchers could hardly eliminate such uncertainty by changing or improving the structural design of an NN, so the end-to-end NN approaches would transmit this uncertainty directly to the final decision which is intolerable in many industrial scenarios. To eliminate such uncertainty, some researchers work hard to ensure that the training set is extensive and comprehensive enough to cover all kinds of situations, but it is too expensive. Moreover, despite the availability of many open-source benchmarks, training datasets with labels are still very scarce in some specific scenarios, such as a fusion vacuum vessel (VV), so achieving consistent and high-quality robotic grasping in such a challenging environment is still quite tricky.

A more efficient approach is to abandon the obsessive pursuit of an end-to-end network and add traditional but controllable approaches between the network output and the final decision, thus eliminating these uncertainties. In our case, we decompose the grasping task into object recognition based on an instance segmentation network and grasping pose computation based on plane extraction and clustering algorithm. In this paper, we chose a well-developed instance segmentation network that has been proposed by Xiang et al. [17] as the fundamental of our robotic grasping methods. Then, we use the clustering algorithm to obtain more stable segmentation results and calculate the correct grasping points and poses via plane extraction. The final experimental results show that our method can achieve a high success grasping rate in the challenging environment.

2. Materials and Methods

The process of our approach is shown in Figure 2. Firstly, the RGB and depth images are entered into the instance segmentation network to obtain a rough result. Then, after filtering these results, we perform cluster to the data, We compare whether the number of categories is the same in ① and ②. ③ represents a pool of data, which stores the latest five frames of data. In ④, we take the intersection of these five frames of data as the final object data. Then, we implement planar extraction to the object data and finally calculate the grasp pose.

2.1. Instance Segmentation Network

As a popular research sub-area in Computer Vision (CV), instance segmentation has been extensively and intensively studied. The Unseen Object Instance Segmentation (UOIS) network approached in [17] is a CNN-based instance segmentation network. It could identify partially obscured objects through the RGB-D messages. In most cases, this method works well for instance segmentation of objects on the table. However, to better simulate the internal environment of the VV, we used a metal tabletop made of the same material as the peg parts, and the 3D point cloud data are quite unstable, as shown in Figure 3.

Thus, we add a clustering algorithm and planar extraction algorithm to obtain a stable grasping result under such a situation.

2.2. Clustering Algorithm

In this paper, we chose Density-Based Spatial Clustering of Applications with Noise (DBSCAN) [18] as the clustering algorithm. As a density-based clustering algorithm, DBSCAN has the advantage of finding arbitrarily shaped clusters in a noisy spatial database. Considering that there are non-negligible errors in the measurement results of depth cameras on non-optimal smooth surfaces, the DBSCAN algorithm is well suited to handle the problem we face. As shown in Figure 2, after we obtain the instance segmentation result from the output of the NN, we then project the 2D information into a 3D point cloud by combining the pixel positions of the segmented instances with the corresponding depth map.The project formulas are as follows:

$$\begin{array}{cc}\hfill & z=Imag{e}_{depth}\left(u,v\right)\hfill \\ \hfill & x=\left(u-C_x\right)\ast z/f_x\hfill \\ \hfill & y=\left(v-C_y\right)\ast z/f_y\hfill \end{array}$$

(1)

u and v are the pixel coordinates in both RGB images and Depth images, $C_x$ and $C_y$ are the coordinates of the camera’s principal point, $f_x$ and $f_y$ are the scale factors of the camera in the u-axis and v-axis directions they could obtain from the camera intrinsic matrix. Moreover, it is vital that RGB images and Depth images are aligned. After that, we perform DBSCAN clustering on the filtered segmented images, as shown in Algorithm 1. When we have accumulated the latest five frames of data, we take the intersection of them as the segmentation result.

Algorithm 1 DBSCAN

Input: dataset $D=\left\{p_1,p_2,...,p_n\right\}$, algorithm parameter $(ϵ,MinPts)$   1: Initializing the set of core points $\Omega =\varnothing$, the number of clusters $k=0$, unvisited sample set $B=D$ and the set of clusters $C=\varnothing$   2: for each $p_i\in D$ do   3:    Initializing its neighborhood points set $N_{ϵ}\left(p_i\right)=\varnothing$   4:    for any $p_m\ne p_i\mathrm{In}\mathrm{addition},p_m\in D$ do   5:      Calculate the distance $d(i,m)$ to $p_i$   6:      if $d(i,m)<ϵ$ then   7:         $N_{ϵ}\left(p_i\right)=N_{ϵ}\left(p_i\right)\cup p_m$   8:      end if   9:    end for 10:    if $\left|N_{ϵ}\left(p_i\right)\right|>MinPts$ then 11:      $\Omega =\Omega \cup p_i$ 12:    end if 13: end for 14: if $\Omega =\varnothing$ then 15:    Quit the algorithm 16: end if 17: Randomly choose an item $o\in \Omega$, initializing the current core points set ${\Omega }_{cur}=\left\{o\right\}$, marking the item as the cluster k, initializing its cluster set $C_k=\left\{o\right\}$ and update $B=B-\left\{o\right\}$ 18: if${\Omega }_{cur}=\varnothing$ then 19:    $C_k$ has been determined, updating the set of clusters $C=\left\{C_1,C_2,...,C_k\right\}$ and back to line 14. 20: end if 21: Updating $\Omega =\Omega -C_k$ 22: Randomly chose an item $o^{\prime }\in {\Omega }_{cur}$, determine its neighborhood points set $N_{ϵ}\left(o^{\prime }\right)$ based on the distance parameter $ϵ$, let $\Delta =N_{ϵ}\left(o^{\prime }\right)\cap B$, updating $C_k=C_k\cup \mathrm{\Delta }$, $B=B-\mathrm{\Delta }$ and ${\Omega }_{cur}={\Omega }_{cur}\cup (\Delta \cap \Omega )-o^{\prime }$, back to line 18 Output: Clusters set $C=\left\{C_1,C_2,...,C_k\right\}$

2.3. Plane Extraction, Grasping Pose Calculation and Contact Point Chosen

Since all the components to be grasped in this application scenario are very regular aggregates, there are very standard planes in these structured data, and the Random Sample Consensus (RANSAC) [19] plane extraction algorithm achieves good results in this case. The RANSAC-based plane extraction in this paper is shown in Algorithm 2. Once the top plane of the object has been extracted, we can easily calculate the grasping pose.

Algorithm 2 RANSAC plane extraction

Input: Point clouds $P=\left\{p_1,p_2,...,p_n\right\}$, initialise the parameter $n=3$, iteration times $k=1000$ and the distance threshold $t=3 \mathrm{m}\mathrm{m}$   1: Initializing $iteration=0$, $Plan{e}_{best}=Null$, $InlierSe{t}_{best}=\varnothing$, $OutlierSe{t}_{best}=\varnothing$, $SumDistanc{e}_{best}=0$   2: for $iteration<k$ do   3:    Randomly pick n points set $P_s$ from P, calculate the $Plan{e}_{cur}$ and initializing $InlierSe{t}_{cur}=\varnothing$, $OutlierSe{t}_{cur}=\varnothing$, $SumDistanc{e}_{cur}=0$   4:    for each $p_m\in \left(P-P_s\right)$ do   5:      Calculate the distance d from point $p_m$ to the $Plan{e}_{cur}$   6:      if $d<t$ then   7:         Updating $InlierSe{t}_{cur}=InlierSe{t}_{cur}\cup p_m$, $SumDistanc{e}_{cur}=SumDistanc{e}_{cur}+d$   8:      else   9:         Updating $OutlierSe{t}_{cur}=OutlierSe{t}_{cur}\cup p_m$ 10:      end if 11:    end for 12:    if $\left|InlierSe{t}_{cur}\right|>\left|InlierSe{t}_{best}\right|$ then 13:      $Plan{e}_{best}=Plan{e}_{cur}$      $InlierSe{t}_{best}=InlierSe{t}_{cur}$      $OutlierSe{t}_{best}=OutlierSe{t}_{cur}$      $SumDistanc{e}_{best}=SumDistanc{e}_{cur}$ 14: else if$\left|InlierSe{t}_{cur}\right|=\left|InlierSe{t}_{best}\right|$ & $SumDistanc{e}_{best}>SumDistanc{e}_{cur}$then 15:      $Plan{e}_{best}=Plan{e}_{cur}$      $InlierSe{t}_{best}=InlierSe{t}_{cur}$      $OutlierSe{t}_{best}=OutlierSe{t}_{cur}$      $SumDistanc{e}_{best}=SumDistanc{e}_{cur}$ 16:    end if 17:    $iteration=iteration+1$ 18: end for Output:$Plan{e}_{best}$, $InlierSe{t}_{best}$, $OutlierSe{t}_{best}$

As shown in Figure 4, the grasping pose has several elements, such as $\mathbf{P},W$ and $\theta$. $\{P_a,P_b,P_c,P_d\}$ are the four corner points of the extracted plane. For each corner point, $P=\{x,y,z,u,v\}$, $\{u,v\}$ are the pixel coordinates of the images, and $\{x,y,z\}$ are the 3D coordinates calculated through Formula (1).

$\mathbf{P}=\{x_c,y_c,z_c,V\}$, $\{x_c,y_c,z_c\}$ are the 3D coordinates of the centre point of the extracted plane, which could easily be calculated. V is the rotation vector which could be calculated as:

$$\begin{array}{cc}\hfill & V_{ab}=(x_b-x_a,y_b-y_a,z_b-z_a)\hfill \\ \hfill & V_{ac}=(x_c-x_a,y_c-y_a,z_c-z_a)\hfill \\ \hfill & V=V_{ab}\times V_{ac}\hfill \end{array}$$

(2)

It should be noted that, according to the right-hand rule, the direction of V is pointing inside the screen, which is in accordance with the pose of the end-effector in the robot’s coordinate system. Since we already have the extracted plane, we can easily obtain the four corner points: $P_a,P_b,P_c$ and $P_d$. Thus, the $\theta$ can be calculated as:

$$\begin{array}{cc}\hfill & \theta =arctan\left(\frac{y_a-y_b}{x_a-x_b}\right)\hfill \end{array}$$

(3)

W is the distance between $P_a$ and $P_c$.

We use the three-finger gripper as the end-effector to grasp the rectangular part of the object, so the three contact points are on the two long sides of the rectangle. The method we propose (called HUDR) is a hybrid one combining UOIS with DBSCAN and RANSAC.

3. Results

3.1. Experimental Platform

Our platform in this experiment in shown in Figure 5. In our case, the top of the peg is made into a rectangular cube—we call it “Thor’s hammer”—to increase the contact area between the gripper and the peg and thus increase the friction, which is beneficial to the subsequent assembly work. We carry out experiments on a desktop with an Nvidia RTX 2080Ti GPU, Intel i7-10700k CPU and 64 GB of RAM. The calibration between the camera and the robot is based on the Tsai–Lenz camera calibration [20] algorithm. The control of the robot and the gripper is implemented through the Universal Robots RTDE package (https://sdurobotics.gitlab.io/ur_rtde/index.html (accessed on 16 June 2022)).

3.2. Visually Intuitive Evaluation

As previously stated, the RGB-D camera has a high level of distortion in the measurement data on smooth surfaces, which affects the output of the instance segmentation network; this phenomenon is also evident in our experiments. Figure 6 shows the results in various stages in the experiment.

The top left picture is the original RGB image. “Thor’s hammer” is the grasping target. The discs with holes on the right are for subsequent assembly tasks and are not relevant to this paper. The picture on the upper right is the result of the segmentation network, where there are obvious errors. We project the classification results from the 2D image into the 3D points cloud and implement the clustering algorithms to obtain the bottom left image, where the top red area is the result of planar extraction. The data obtained after the planar extraction are displayed independently in the lower right image. From the pictures, we can see that the recognition results are greatly improved after our algorithm.

3.3. Quantitative Evaluation

In this subsection, we will compare the success grasping rate of our method with the GR-ConvNet and UOIS methods. The GR-ConvNet is an end-to-end grasping network that outputs a rectangle representing the grasping pose and width. UOIS is the fundamental instance segmentation network of our method. In the experiment, we calculate the grasping success rate of the UOIS method by calculating the grasping pose directly from the segmentation results output by the network.

To better simulate the environment in VV, the object to be grasped was placed on a metal table in this experiment, as shown in Figure 5. To compare the effect of different materials of the background environment, we also conduct comparative experiments on wooden tables. We performed 100 grasping operations each with these methods under the same scenarios and then calculated the success rate. These scenarios include two different poses: standing pose, lying down pose and two different backgrounds: wooden table and metal table, as shown in Figure 7.

The results in

Table 1. Success rate comparison.

	GR-ConvNet	UOIS	HUDR
wooden table + lying pose	78%	86%	93%
wooden table + standing pose	53%	92%	100%
metal table + lying pose	49%	44%	82%
metal table + standing pose	26%	71%	97%

show that the different backgrounds make a big difference in the results. If the background and the object are very close in material and color, it is difficult for these networks to identify the object correctly. Meanwhile, the performance of the GR-Convent for the standing poses is much poorer than the lying ones. This is because, when the object is standing on the table, the height difference of the object becomes more significant, and the error in the depth information from the RGB-D camera becomes more prominent, especially for the cylindrical part at the bottom. On the other hand, GR-Convent is more likely to choose the cylindrical part as the grasping target, which means it could hardly find an appropriate grasping pose.

However, the UOIS and the method proposed in this paper are based on instance segmentation. When the object is in its standing pose, the instance segmentation network in these methods could better identify the top plane of the object. Thus, this is why these methods’ performance on the standing pose are better than the lying pose.

Another factor that influences the proposed method’s performance on the lying pose is that, when the object is lying down, the sides of the cylinder would have some adverse effects on the clustering algorithm.

Overall, our method performed much better under different scenarios. In particular, our method still achieved a 97% success grasping rate in the closest conditions to the actual situation (metal table plus standing pose). Most importantly, when our algorithm does not perform well, we can find which part goes wrong and improve it. For our method, the failure cases during the grasp are because of the error of the clustering results. We can adjust the parameters $ϵ$ and $MinPts$ for different situations. The tuning of these parameters can intuitively change the performance of the algorithm for different cases; for example, if we want the algorithm to be more conservative, we also reduce the value of $MinPts$. It makes our approach more reliable and controllable in this challenging environment.

4. Conclusions

In this paper, we focus on robotic grasping under a challenging environment in which the data of the RGB-D camera exhibit significant distortion. To eliminate the uncertainty of the grasping result, we proposed a method which combines the CNN-based instance segmentation network with clustering and the plane extraction method. The experimental results fully demonstrate the superiority of our approach in this highly challenging environment. Meanwhile, our approach lacks universality, so our future work will focus on improving the robustness of our algorithms while maintaining stability and high success rates and on carrying out subsequent studies on the peg-in-hole assembly task.