The Design and the Development of a Biped Robot Cooperation System

Abstract

The aim of this paper is to design a fuzzy motion control algorithm for a developed monocular vision system based on a cooperative transportation system of two humanoid robots. The control strategies of the cooperation transportation system contain three stages, including object searching, walking toward the transported object, and cooperatively moving the transported object. To have different moving speeds, the gait step size was pre-planned as two different modes, i.e., one of the gaits is selected to let the HR have large variations of motion and another gait is to make the HR with small variations. The fuzzy motion control algorithm is utilized to select the appropriate mode of gait. Both humanoid robots can actively search and move to the front of the target object, then cooperatively lift the target and carry it to the platform. The task of synchronous movement is controlled with fuzzy techniques through the control terminal. From the experimental results, it can be seen that both robots can distinguish the orientation of the target, move to the appropriate position, and then successfully raise the target together.

1. Introduction

In recent years, the application of robots has become increasingly widespread due to the development of control technology and artificial intelligence, especially in the area of the cooperation of multi-robots [1,2,3]. Applications for multiple robot systems include assembly and painting of industrial manipulators [4,5], transportation and formation of multi mobile robots [6,7], crop harvesting of a mobile manipulator [8,9], and hybrid collaborative robot systems [10,11].

Humanoid robots are more flexible than other types of robotic systems since their firm multi-joint structure can complete more sophisticated and tedious tasks [12,13]. Especially, humanoid robots can be applied for object transportation, rescue tasks, sports competition, and logistics handling cooperation [14,15,16,17]. However, a single humanoid robot carrying a heavy object might lose the chance to complete the task successfully when the weight of the heavy object is too large. Therefore, the problem of multi-humanoid robot cooperation has been becoming a hot topic in the field of robotics [18,19,20].

In [18], the research focused on the cooperation between two robots, which is named NAO. However, the behavior of collaboratively carrying objects is based on the way of interaction between human and human. Two robots can achieve synchronous control by multithread communication under the cloud platform models. In [19], based on multi-sensor fusion, kinematics modeling, and object recognition, the authors analyze and design the dynamic identification of object position and pose, trajectory planning of robot motion, and cooperative handling control. Two NAO robots can cooperatively accomplish the task of transportation using the proposed strategies in [19]. In [20], a dynamic model that satisfies the cooperative movements is established, and the motion trajectory of two humanoid robots in the process of cooperative manipulation of objects is planned. Furthermore, the control method based on optimal parameters is adopted to optimize the energy consumption of the cooperative movements of two humanoid robots.

Since the exact model of humanoid robot is hard to obtain, the control method in [20] is not easy to implement in a real humanoid robot. Although the control strategy does not rely on the exact model of a humanoid robot in [19], more sensors, including a Zed2i binocular stereo vision sensor, a RealSense D455 deep infrared sensing camera, and several built-in sensors, are required in a humanoid robot. In [18], the distance between the object and the target is calculated using only one vision sensor. Additionally, the mathematics model is unnecessary in [18]. Moreover, that the object is cooperatively transported to the specified location by two robots is not really implemented in [18].

In this paper, a cooperative transportation system with two humanoid robots is developed. Let the humanoid robots (HRs) could walk smoothly and quickly to near the transported object (TO), a monocular vision system based on fuzzy motion control algorithm is proposed. In general, the processing time of vision analysis is large when the algorithm is implemented in a small single-board computer such as Raspberry-Pi-2. To reduce the computational complexity, the gait step size is pre-planned as two different and fixed modes, i.e., one of the gaits is selected to let the HR have a large variation of motion and another gait is to allow the HR a small variation. The fuzzy motion control algorithm is utilized to select the appropriate mode of gait. There are two stages, including the processing of image information and motion control, in the one period control process. The control performance would be affected by the large image processing time. Thus, reducing the unnecessary image processes would increase the time for the control stage to improve the control performance. Compared with other research of multi-HRs cooperative control, the contributions and novelties in this paper are summarized as follows.

• This paper proposes a study to design a biped robot cooperation system, since few literatures discuss the collaborative transportation of two humanoid bipedal robots;
• The proposed fuzzy motion controllers are not dependent on the mathematical model of humanoid robots. Furthermore, the fuzzy motion control algorithm is utilized to select the appropriate mode of gait to have a different moving speed;
• The HR can capture the environment information only using a monocular vision sensing system. Moreover, the computational complexity of the proposed algorithm is moderate such that both the image processing and the motion controller can be implemented on RPi-2.

The remainder of this paper is organized as follows. Section 2 describes the architecture of a biped robot cooperation transportation system. The motion control algorithms are developed in Section 3. Section 4 presents an experiment to verify the effectiveness of the proposed method. Finally, conclusions are presented in Section 5.

2. Architecture of Biped Robot Cooperation Transportation System

2.1. The Configuration of the Robot System

The diagram of the developed cooperation transportation system is shown in Figure 1. Two humanoid robots are considered in the developed system. Each robot includes a Raspberry Pi-based (RPi) vision system and a modified commercially available humanoid robot XYZ Bolide Y-01 (BHR). The BHR includes eighteen smart servo motors (A1-16) with a high torque ratio (max. torque 25 kg-cm) in which each hand has three degrees of freedom (DoFs) using three motors and each leg has six DoFs using six motors. All smart servo motors are controlled by an Arduino-compatible ATmega1280 microcontroller (RC2200). The RC2200 control board includes a three-axis accelerometer and a distance sensor.

Compared with the original commercially available BHR, there are two parts modified in this study, including a non-functional head that is replaced by a RPi-based vision system, and a smaller grasper is changed to be a longer homemade palm. The modified BHR has the following dimensions L: 140 mm, W: 230 mm, H: 470 mm. The RPi is utilized to capture and to analyze the image of the transported object (TO). In this study, the modified BHR can capture environmental information using only the developed RPi vision system. Moreover, the calculation of the appropriate motion and the message exchange between robots are processed in RPi. Two BHRs exchange information by the socket communication via Wi-Fi.

2.2. Image Processing

In this research, a white rectangular box is selected as a TO with a red marker attached to the corresponding sides of the box as shown in Figure 2a. The HRs can find the TO using the vision system. The vision system, which is placed on the head of the HR is shown in Figure 1, includes a Microsoft HC-3000 Webcam, a SG-90 servo motor, and Raspberry-Pi-2. The webcam is mounted on the SG-90 servo motor such that the field of view (FOV) can be maximized to $\pm 180°$ by rotating the SG-90.

The purpose of the image processing is to have the HR be able to find the TO. Moreover, the HR can obtain the relative orientation from the red masker and then move to the front of the TO. The flowchart of the image processing is illustrated in Figure 3. In the initialization stage, the servo motor would be returned to make the camera face to the front of the HR. Then, the vision system captures an image. This captured image is smoothed with the Gaussian filter to reduce the noise.

In general, the captured image is in 8-bit red-green-blue (RGB) format. The three elements of color image in RGB are all co-related with the amount of light hitting the object. It is hard to separate color information from luminance. In other words, we cannot easily detect the correct color information of the red marker when the luminance of the environment is varying. Therefore, we convert the image format from the RGB model to hue, saturation, and value (HSV) model. “Hue” represents the color, “Saturation” represents the amount with which that respective color is mixed, and “Value” represents the amount of black with which that respective color is mixed. HSV is used to separate image luminance from color information. HSV is more robust external lighting changes. The image processing techniques that are used in this study are implemented by OpenCV and Python on RPi. The kernel dimension of the Gaussian filter is chosen $5\times 5$ in the practice experiment.

2.3. Orientation Calculating

The vision system further detects whether the red marker is found in the FOV. When the red marker is not found, the SG90 servo motor will rotate a pre-defined angle such that the robot can search for the red marker with a different region until the red marker of the TO is found. Due to the fact that the HR has only a vision sensor to obtain the orientation information of the TO, it is important to find an approach to let the HR move to the front of the TO with the insufficient image information. The captured image is shown as Figure 2b when the HR is standing on the front of the TO. When the TO is in the left front of HR, the view of the HR is shown in Figure 2c. Figure 2d is shown as the TO is in the right front of HR. First, we need to find the corners of the red marker. The center point (the yellow point in Figure 2) and the center of gravity (the blue point in Figure 2) can be calculated using four corner points. Then, we can obtain the orientation information of TO using the relationship between the center point and the center of gravity point (CoG). The center point, ${\mathit{P}}_c={\left[x_c,y_c\right]}^T$, can be calculated as follows

$${\mathit{P}}_c=\frac{1}{4}{\displaystyle \sum }_{i=1}^4{\mathit{P}}_{corner}^i,$$

(1)

where ${\mathit{P}}_{corner}^i={\left[x_{corner}^i,y_{corner}^i\right]}^T,i=1,2,3,4$ is the i-th corner coordinate (the red point in Figure 2). The center of gravity, ${\mathit{P}}_{cg}={\left[x_{cg},y_{cg}\right]}^T$, is calculated in the following

$${\mathit{P}}_{cg}=\frac{1}{n}{\displaystyle \sum }_{i=1}^n{\mathit{P}}_{red}^i,$$

(2)

where ${\mathit{P}}_{red}^i={\left[x_{red}^i,y_{red}^i\right]}^T,i=1,\dots ,n$ is the coordinate of the i-th points of the red marker. Therefore, the orientation of the TO in view of HR can be represented as

$$\{\begin{array}{c}centerfront\left(CF\right),if\left|x_{cg}-x_c\right|<\epsilon \\ rightfront\left(RF\right),if{x}_{cg}-x_c>\epsilon \\ leftfront\left(LF\right),if{x}_{cg}-x_c<-\epsilon \end{array},$$

(3)

where $\epsilon$ is a constant.

2.4. Distance Estimating

As previously mentioned, the sensing ability of HR depends only on the vision system. Besides the orientation, the distance between the HR and the TO, $d_{TO}$, is another important information. It can be easily seen that HR can lift the TO correctly only when the following condition is satisfied:

$$\underset{\_}{d}\le d_{TO}\le \overline{d},$$

(4)

where $\underset{\_}{d}$ and $\overline{d}$ are constants corresponding with the arm length of the HR, and $\underset{\_}{d}<\overline{d}$.

Additionally, the mode of gait pattern of the HR is selected according to the $d_{TO}$. There are two stages, including the processing of image information and motion control, in the one cycle of the control process. Therefore, the processing time, $t_p$, in one cycle of the control process can be represented as

$$t_p=t_{image}+t_{motion}+t_{idle},$$

(5)

where $t_{image}$ is the time of image processing, $t_{motion}$ is the execution time of completing one gait, and $t_{idle}$ is the idle time and $t_{idle}\ll \min \left\{t_{image},t_{motion}\right\}$. It is not efficient that the gait is fixed in a fixed $t_p$ due to the limited computational capability of the vision system. Therefore, if the distance is far, $d_{TO}\gg \overline{d}$, the gait of the HR can execute repeatedly more times such that the period time can be rewritten as

$$t_p\left(d_{TO}\right)=t_{image}+n\left(d_{TO}\right)\times t_{motion}+t_{idle},$$

(6)

where $n\left(d_{TO}\right)$ is a function of $d_{TO}$ to represent the repeat number and the detail of $n\left(d_{TO}\right)$ is described in Section 3. Since the size of the marker on the box can be known in advance, we can have the real width, $w_p$, of the TO. In addition, the distance information $d_{TO}$ can be calculated with the focal length of the camera, $f$,

$$f=\frac{w_p{d}_{TO}^{\prime }}{w_{TO}},$$

(7)

To find the $f$, the HR and the TO are placed on a preset position with a known distance, $d_{TO}^{\prime }$, between the HR and the TO. Then, with the vision camera opened, the width $w_{TO}$ of the TO image can be determined when the distance between HR and TO is fixed at $d_{TO}^{\prime }$. Since the focal length of the camera is found, the distance of the object to the camera can be calculated as

$$d_{TO}=\frac{w_{TO}f}{w_p},$$

(8)

when the HR starts to move. Although the distance that is obtained by Equation (8) is not precise, it is still necessary and valuable information to have the HR move to the object and satisfy the condition in Equation (4).

3. Cooperative Transportation Strategy

The control strategies of the cooperation transportation system contain three stages, including object TO searching, walking toward the TO, and cooperatively moving the TO. In the first stage, each HR would search for the red marker on the TO and turn to face the TO. In the stage of walking toward the TO, a motion controller is designed to let condition (4) be satisfied. Finally, two HRs would cooperatively move the TO to the destination after the second stage of each HR is completed. Meanwhile, the motion calibration would be triggered to avoid the TO falling when condition (4) is not satisfied in either HR. A flowchart of the control strategy is shown in Figure 4.

3.1. Fuzzy Technique

The motion controllers in various stages are based on fuzzy techniques and two distinct pre-planned gaits. As aforementioned in Section 2.4, the selected gait of the HR can execute repeatedly more times in one cycle of the control process to improve the control performance. Different repeat numbers would be adopted in the different control stages. The repeat number, $n_R\left(\theta \right)$, is chosen according to the necessary turned angle in the stage of object searching and the repeat number, $n_W\left(d_{TO}\right)$, is dependent on the distance $d_{TO}$ in stage of walking toward the TO. In this research, the fuzzy technique is utilized to calculate the appropriate repeat number in each stage of motion control.

Consider $r$ fuzzy rules as

$$R^j:\mathrm{IF}\xi \mathrm{is}{M}^j\mathrm{THEN}\kappa \mathrm{is}{G}^j,j=1,2,\dots ,r,$$

(9)

where $R^j$ is the j-th rule, $\xi \in \left\{\theta ,d_{TO}\right\}$ and $\kappa$ are the input and output of fuzzy system, respectively. $M^j$ and $G^j$ are the fuzzy sets characterized by fuzzy membership function ${\mu }_M^j\left(\xi \right)$ and ${\mu }_G^j\left(\kappa \right)$, respectively. In this research, the triangular membership functions are selected for ${\mu }_M^j\left(\xi \right)$ and the singleton membership functions are chosen for ${\mu }_G^j\left(\kappa \right)$, shown as Figure 5. By using the centroid defuzzification technique, the output of fuzzy algorithm can be obtained as

$$\kappa ={\displaystyle \sum }_j{g}^j{\mu }_M^j\left(\xi \right).$$

(10)

As in Figure 5, $m^j\in \left\{{\beta }_1,{\beta }_2,\dots ,{\beta }_{\gamma },\dots ,{\beta }_r\right\}$ is the parameters of triangular membership function in the input-part of a fuzzy controller and $g^j\in \left\{w_1,w_2,\dots ,w_{\gamma },\dots ,w_r\right\}$ is the parameters of singleton membership function in the output-part of a fuzzy controller.

3.2. Object Searching

The aim of this stage is to let the red marker be found by the vision system in the FOV. The SG90 servo motor will rotate at a pre-defined angle until the red marker of the TO is found. The accumulative rotating angle of SG90 is saved as $\theta$ when the red marker of the red marker is found. Then, the HR needs to be controlled to turn an angle $\theta$.

In general, HR cannot turn to the desired angle $\theta$ in one move. We design two different and fixed modes of turning gait, i.e., one of the turning gaits is selected to let the HR have a large variation of turning angle if $\left|\theta \right|>{\theta }_{th}$ and another gait is to allow the HR with a small variation if ${\theta }_{th}>\left|\theta \right|>\epsilon$, where ${\theta }_{th}$ is a switching threshold. Based on the fuzzy technique, an algorithm to determine the turning repeat number is designed as follows

$$R^j:\mathrm{IF}\theta \mathrm{is}{M}^j\mathrm{THEN}\kappa \mathrm{is}{G}^j,j=1,2,\dots ,5,$$

(11)

and the parameters of fuzzy system are selected as $m^j\in \left\{-\pi ,-{\theta }_{th},0,{\theta }_{th},\pi \right\}$ and $g^j\in \left\{-5,-2,0,2,5\right\}$. The repeat number, $n_R\left(\theta \right)$, is calculated in the following

$$n_R\left(\theta \right)=\min \{n\in \mathbb{Z}|\left|\kappa \right|\le n\}.$$

(12)

The turned direction is left if $\mathrm{sgn}\left(\kappa \right)$ is negative and right if $\mathrm{sgn}\left(\kappa \right)$ is positive. The pseudocode of object searching is shown in Algorithm 1.

Algorithm 1: Algorithm of searching TO.
1:	Initialization: setting the SG-90 to the home position
2:	if the red market is not in front of FOV then
3:	if the red marker is in the left front of FOV then
4:	The robot should turn left
5:	elseif the red market is in the right front of FOV then
6:	The robot should turn right
7:	else
8:	The robot should turn left
9:	Goto 2
10:	end
11:	end

3.3. Moving to Object

To control the HR walking to an object, we designed two different and fixed modes of moving gait, i.e., one of the gaits was selected to let the HR have a large variation of motion if $d_{TO}>d_{th}>\overline{d}$and another gait was to allow the HR a small variation if $d_{th}>d_{TO}>\overline{d}$ or $d_{TO}<\underset{\_}{d}$, where $d_{th}$ is a switching threshold. Based on the fuzzy technique, an algorithm of a moving repeat number is designed as follows

$$R^j:\mathrm{IF}{d}_{TO}\mathrm{is}{M}^j\mathrm{THEN}\kappa \mathrm{is}{G}^j,j=1,2,\dots ,5,$$

(13)

and the parameters of a fuzzy system are selected as $m^j\in \left\{\overline{d},1.5\overline{d},d_{th},4\overline{d},10\overline{d}\right\}$ and $g^j\in \left\{0,1,2,3,5\right\}$. The repeat number, $n_W\left(d_{TO}\right)$, is calculated in the following

$$n_W\left(\theta \right)=\min \{n\in \mathbb{Z}|\kappa \le n\}.$$

(14)

The pseudocode of walking to TO is shown in Algorithm 2.

Algorithm 2: Algorithm of walking to TO.
1:	if $\underset{\_}{d}\le d_{TO}\le \overline{d}$ then
2:	go to the next stage
3:	end
4:	if $d_{TO}<\underset{\_}{d}$ then
5:	HR go back with small variation gait
6:	elseif $d_{th}>d_{TO}>\overline{d}$ then
7:	HR goes forward with small variation gait and repeat $n_W\left(\theta \right)$
8:	elseif $d_{TO}>d_{th}>\overline{d}$ then
9:	HR goes forward with large variation gait and repeat $n_W\left(\theta \right)$
10:	end
11:	go to the stage of transported object searching

3.4. Cooperative Transportation and Calibration

In the stage of cooperatively moving the TO, the movements of the two HRs must be synchronized. Therefore, a master–slave system using TCP/IP to communicate through a wireless network is implemented to exchange the movement command between the two cooperative HR. The master HR would transmit a motion request to the slave HR and await the Acknowledge (ACK) from the slave HR. Then, the master HR executes the according motion when the ACK is received. On the other hand, the slave HR would transmit an ACK to the master HR when it receives a request, and it has completed the previous action. After ACK is transmitted, the slave HR would execute the specified motion. In this stage, to complete the task that moves the TO from the starting position to the desired position, the candidate motions contain raising arm, walking sideways, and putting down.

Since the type of HR arm is a tray, not a gripper, the TO might be dropped during transportation. In general, the motion of walking sideways is executed to make not only lateral movement but also slightly turning such that the accumulated turning angle would be increased. The HR moving sideways with the existing turned angle would lead the distance $d_{TO}$ to be increased or decreased such that condition (4) fails. Therefore, the angle of facing the TO needs to be calibrated. The rotating angle of SG90 cannot be utilized to estimate the facing angle because the red marker is always in the FOV, when $d_{TO}<\overline{d}$. To overcome this problem and to estimate the relative orientation between the HR and the TO, the center point and the center of gravity point, which are described in Section 2.3, are adopted to calculate the orientation in Equation (3). The process of calibration would be enabled and the process of moving the TO would be paused, when the condition $\left|x_{cg}-x_c\right|<\epsilon$ was not satisfied. Then, the HR would be controlled by the corresponding motion according to Equation (3) until the conditions $\left|x_{cg}-x_c\right|<\epsilon$ and $\underset{\_}{d}\le d_{TO}\le \overline{d}$ were satisfied. The pseudocode of cooperative transportation and calibration is shown in Algorithm 3.

Algorithm 3: Algorithm of cooperative transportation.
1:	Awaiting the condition (4) is satisfied in each HR
2:	Lifting TO
3:	whileTO does not arrive the desired position do
4:	if$\left|x_{cg}-x_c\right|>\epsilon$ then
5:	repeat
6:	motion calibration
7:	until$\left|x_{cg}-x_c\right|<\epsilon$ and $\underset{\_}{d}\le d_{TO}\le \overline{d}$
8:	end
9:	HR walking sideways
10:	End
11:	Putting down TO

4. Experimental Results

Two experiments are presented to validate the effectiveness of the proposed approach in this section. The first scenario is that one HR moves toward the TO with/without the proposed fuzzy motion controller. The second scenario is that two HRs cooperatively transport the TO. Due to the load capacity of HR, a white box, in which its dimension is (L: 320 mm, W: 220 mm, H: 110 mm) and its weight is 200 g, is selected as a TO in each experiment. To facilitate the observation and recording of the experimental results, an additional red marker is placed on the top of the white box, except on both sides. Since the performance of the smart servo motor would be affected by a low battery, the power source of each HR is an external power adapter to guarantee that the supply voltage can be stable in the experiments.

The first experiment aims to compare the response of motion control with and without a fuzzy technique. The experiment results are illustrated in Figure 6 and Figure 7, where Figure 6 and Figure 7 are the snapshots of the HR moving toward the TO without and with a fuzzy technique, respectively. In this experiment, the HR is initially placed approximately 60–80 cm away from the TO. From Figure 6, it is seen that it takes approximately 66 s for the task to be accomplished, with the proposed motion controller without fuzzy technique. However, the lifted box looks a little askew at the end of the experiment. From Figure 7, we can obviously see that it takes only 55 s to precisely lift the TO with the proposed fuzzy motion controller. To verify its reproducibility, the same experiments are repeated ten times. Except the proposed fuzzy motion controller has a shorter execution time; the situation that the lifted box looks a little askew only occurs one time with the proposed fuzzy motion controller but four times using the motion controller without fuzzy technique.

The second experiment is to validate the effectiveness of the proposed fuzzy motion controller for two HRs to cooperatively transport the TO. The configuration of the second experiment is shown in Figure 8, where the white rectangle is the initial location for the TO, the target location of the TO is the gray rectangle. Each black circle represents the HR and the corresponding initial orientation. It is noted that two HRs are placed away on both sides of the box. The initial orientation of HRs is set to have each HR not face the object. Each gray circle represents the target position of each task, respectively. That is, one is the target position for the task to move toward the TO. Another is the target position for the task of moving the TO to its desired position. The dash line is shown as the desired moving trajectories of the two HR. The objective of the experiment is that two HRs cooperatively transport a TO from the initial location to the desired location. The distance between the HR and the TO is approximately 40–60 cm and the distance between the TO and the desired position is also approximately 30–40 cm.

The experimental results are illustrated in Figure 9 (including fourteen subfigures). Two HRs need to cooperatively transport a box to be on top of the white brick in the left. From 1 to 3 in the Figure 9, we can see that two HRs can turn to face box first. Then, each HR moves towards the TO. The HR will lift the TO when distance $d_{TO}$ is smaller than $\overline{d}$, which is shown in 4 and 6. Figure 7, Figure 8 and Figure 9 show two robots that cooperatively transport an object moving laterally to the desired position. Finally, two HR can put the TO down gently on the white brick, which is shown in 12. Similarly, to verify the experiment’s reproducibility, this experiment is repeated ten times. Both HRs can move to the front of the TO at all times, and HRs can put the TO steadily on the platform at the desired position with a success rate of 0.8; however, HRs can put the TO at the desired position without a platform with a success rate of 0.9.

5. Conclusions

A cooperative transportation system with two humanoid robots has been designed in this paper. A raspberry Pi-based vision system has been developed on each modified commercially available Bolide Y-01 humanoid robot. The motion controllers in various stages are based on fuzzy techniques that are utilized to calculate the appropriate repeat numbers in each stage of the motion control. The proposed fuzzy motion controllers are also successfully implemented on the RPi to complete the transportation tasks. From experimental results, it is noted that each robot can search the TO and then move the TO from the starting position to the destination by the proposed algorithm, using only vision feedback. The proposed method can be applied to the simple and the cheap humanoid robot system to execute the cooperative tasks. In this study, the TO is assumed to be well-known, including its shape, dimension, and suitable grab points. In future work, we will further study how to transport a non-rectangle object, especially without a specific marker to let the proposed biped robot cooperation system to be applied to practical applications.