Prototype of 5G Integrated with TSN for Edge-Controlled Mobile Robotics

Abstract

The digitization of industries enables a rapid transformation from mass production to individualized manufacturing. Communication plays an essential role in this digital transformation; in particular, wireless communication enables a high degree of flexibility, dynamic interactions, and mobility support in production systems. This paper presents an implementation of a 5G system with Time-Sensitive Networking (TSN) and analyzes a typical industrial use case involving cloud-controlled mobile robots. A prototype setup integrating 5G in a TSN network has been completed to evaluate the 5G-TSN performance for industrial applications. The integrated 5G and TSN prototype has been evaluated with over the air tests in an industrial shopfloor using TSN features of traffic shaping and scheduling.

1. Introduction

A large-scale communication network is a crucial aspect of smart factories. It not only enables communication among actuators and sensors, enabling an adaptive and flexible process, but also an integration of large computing resources such as edge cloud and factory cloud systems. Factory cloud refers to a local server system near the shopfloor with cloud capabilities. Artificial intelligence, big data analysis, and cloud-based control systems increase production efficiency and quality by enabling overall orchestration and optimization. Combined with wireless communication, the digitalization of production systems can achieve high flexibility, allowing customized products with a batch size of one. Especially autonomous guided vehicles and mobile robots can benefit from wireless communication and cloud systems. Moving path planning and control algorithms to a factory cloud system enables smart and collaborative decision making and fast reaction without efficiency loss.

Industrial manufacturing communication needs to meet rigorous requirements regarding reliability, availability, and real-time capability [1,2]. State-of-the-art real-time communication is dominated by fieldbus protocols, such as EtherCat, PROFINET/PROFIBUS, or CC-Link. These protocols were designed for local connections inside a machine or a production cell with the focus on the high reliability and short cycle times. As a result, they are mostly manufacturer-specific and at the most compatible on the physical layer. Furthermore, all these solutions for industrial communication are wired based, creating static connections. Wireless solutions have only been used for a limited number of applications in industry, which are caused by the high requirements in the production environment [3,4]. Current state-of-the-art solutions such as Wi-Fi or Bluetooth cannot serve these requirements; therefore, communication networks are limited to wired communication. In consequence, industrial networks are rigid and do not provide the flexibility and connectivity needed for Industry 4.0. Two main deficits have been identified: Existing communication solutions are vendor specific and therefore not compatible with each other, and they do support wireless solutions.

With IEEE 802.1 Time-Sensitive Networking (TSN) [5], a set of standards for deterministic communication has been introduced to IEEE 802.1 and IEEE 802.3 to support real-time communication over Ethernet networks. With TSN, large-scale Ethernet networks can transmit real-time critical data suitable for industrial applications [6,7]. This reduces the need for specific fieldbus protocols and vendor-specific communication systems. TSN introduces three main areas to Ethernet: time synchronization, sending scheduled traffic, and centralized, automated system configuration [8]. In the prototype, TSN is used for the wired connection of the field devices with the factory cloud. Owing to its compatibility with enterprise networks, its vendor independence, and its scalability, TSN is well suited for large-scale networks for production in future.

The 5G system is seen as a key enabling technology for wireless communication in industrial use cases. Meanwhile, 3GPP, the standardization body for 5G, has made design considerations from the very beginning to define technology features that allow ultra-reliable low-latency communication (URLLC) for industrial use cases [9,10]. The 3GPP Releases 15 and 16 specify various technology features to enhance reliability and provide low-latency communication, including faster signaling schemes, traffic prioritization, redundancy and robustness for control and data transmission, etc. [11]. Furthermore, 3GPP has specified the technological framework to allow the integration of 5G with TSN [12,13,14]. The basic functionalities and features to allow 5G integration with TSN have been completed in 3GPP Releases 16 and 17. The 5G system is treated as a TSN bridge in the 5G-TSN integration framework. Features such as native support for Layer-2 (Ethernet) traffic, efficient transportation of TSN Ethernet traffic with compression schemes in the 5G network, End-to-End (E2E) time synchronization, Quality of Service (QoS) differentiation, priority class handling to map QoS flows of TSN traffic to 5G QoS, and translation functionalities for 5G-TSN interworking are supported. The 5G-ACIA (5G Alliance for Connected Industries and Automation) industrial forum has been investigating the detailed requirements, interfaces, and architectures that allow the integration of 5G with TSN-enabled networks in industrial use cases, bringing both automation industry and information communication technology players together [15].

We have carried out a prototype setup integrating 5G in a TSN network to evaluate 5G-TSN performance for industrial applications, mainly in real-time analysis of the communication. The industrial application requirements are defined for virtual Programmable Logic Controllers (PLC) in mobile robotics use cases communicating with cyclic and synchronous signals. This prototype setup consists of a 5G URLLC standard-compliant pre-commercial system and a wired network supporting different TSN standards. The integration of the 5G URLLC prototype system has been carried out as a virtual bridge between the TSN switches as described in [15].

The structure of this paper is as follows: Section 2 describes the mobile robotics use case and its requirements toward the communication infrastructure. In Section 3, the architecture and implementation of the 5G-TSN communication prototype are given. The detailed performance measurements carried out on the integrated prototype system for the use case requirements and a discussion of the results are presented in Section 4. Section 5 summarizes the paper and provides an outlook.

2. Use Case Mobile Robotics and Requirements

Mobility is an essential requirement of flexible lineless assembly systems to meet the market demand for highly customized products. Recent advancements in robotics, sensor systems, and communication infrastructure push toward ultra-flexible production systems [16]. Industrial mobile platforms enable temporary production cells in assembly stations. Mobile robots enable new sets of use cases, and to realize them, reliable real-time communication is a must for mobile robots. Reliable real-time communication between the mobile robots and the factory cloud system enables collaborative decision-making tasks and advanced AI/ML methods, which was impossible with onboard processing on the mobile robots. In the simplest case, a mobile robot consists of a power supply, a platform, a manipulator, a computing and control unit, and several sensors. Depending on the applications and use cases, the mobile robot can be equipped with specific components, such as a loading unit or conveyor belt for transport. Mobile robots for more complex industrial applications are often equipped with a manipulator, enabling several possible assembly tasks such as pick and place, handling (storage, placing, holding, setting, etc.), and supporting processes (marking, cleaning, inspection, etc.). Recent publications [17,18] highlight the following advantages of mobile robots compared to fixed robots:

• Greater reconfigurability and flexibility of the production system;
• Increased production efficiency with high product variability;
• Greater reliability and less downtime.

2.1. Factory Cloud-Based Control of Mobile Robotics

In cloud robotics, the control functionality of the robot is offloaded from the robot platform into an edge-cloud computing platform, such as a factory cloud, as shown in Figure 1. The control functionality includes motion planning, PLC, localization and mapping, and all the monitoring and computational demanding processes. The robot controller in the factory cloud computes the trajectory and predicts how to act in the environment with obstacles. This decision is based on the data that a mobile robot transmits from internal and external sensors to the factory cloud system for processing via wireless communication [19]. The evaluation and accuracy of the data are strictly connected with the choice of sensors. There are many approaches for data evaluation, since the correct and quick perception of the environment is seen as the most important characteristic of a mobile robot. Especially, approaches from computer vision and deep learning in connection with various camera-based sensors have shown a high potential for this.

Although mobile robots have great potential for many applications, they are still at the beginning of development and have only found their way into production chains in isolated cases. This is, among other things, due to some important challenges of current mobile robots. The most important challenges are listed below [20]:

• Localization, navigation, and trajectory planning are highly complex and are still the subject of current research;
• Up to now, safe cooperation with people has only been possible in restricted movement modes;
• Detection of obstacles is not possible in the required time, depending on the sensor technology used.

The limitation of the safe operation of a robot in a production environment is usually not specified by the moving mass but by the speed, detection, and reaction time. However, the sophisticated algorithms for obstacle detection and reaction often demand extensive computational resources that are generally unavailable on mobile robots, e.g., for energy consumption considerations. To establish complex anatomies, the decentralized processing of data acquired by the mobile robots enables high flexibility and supplies the computational resources required for complex tasks but has not been possible in the past due to the latency and reliability limitations of conventional wireless communication systems. As a new wireless technology, 5G is designed to satisfy both low-latency and high-reliability communication demands and provides enough bandwidth to resolve the issues mentioned above.

Both 5G and TSN enable the real-time connectivity of mobile robots to the factory cloud systems, and with that, mobile robots can transfer computationally intensive tasks to the factory cloud systems [21]. By offloading such tasks from mobile robots, sophisticated and complex algorithms are possible for the localization tasks, safe cooperation, and real-time obstacle detection on the factory cloud systems. In this research, the cooperation between two mobile robots is targeted to be analyzed. Both 5G and TSN play a crucial role in mobile robots’ synchronized movement when picking the material cooperatively in this use case.

2.2. Requirements toward the Communication Infrastructure

In addition to wireless communication, mobile robots need reliable and deterministic communication. In this use case, the robot has a LiDAR (Light Detection and Ranging) system for detecting the environment, a camera system for precise positioning of the tool, and a cloud-based PLC. These three different data streams have different requirements regarding reliability and real time.

To define the requirements, the traffic types and characteristics given in [22] and 5G-ACIA are taken as reference, and the requirements are extended based on our industrial use case.

Table 1. Traffic characteristics for the mobile robotics use case.

	Cloud-Based PLC	LiDAR	Camera Stream
Traffic Type	Cyclic	Events: alarm and	Video
	Asynchronous	Operator control
	Periodic	Sporadic	Periodic
Period	7 ms	25–100 ms	15–30 fps
Data Delivery Guarantee	Latency Bound <Period 90%	Latency Bound Min. Throughput	Throughput
Tolerance to Loss (Survival Time)	1–4 frames (Loss of 1 to 4 consecutive frames)	Yes	Yes
Data Size	Fixed 50–80 Bytes	Variable 100–160 Bytes	Variable >1000–5000 Bytes
Criticality	High	Medium	Low
	Hard Real-Time	Soft Real-Time

shows the traffic profiles for our use case. The Data Delivery Guarantee in the table indicates the aspect of the traffic’s requirement which needs to be guaranteed to make sure the traffic does not experience any errors or interruptions. In a cloud-based PLC, Data Delivery Guarantee indicates the maximum bound of communication latency between the transmitter and receiver to make sure the connection is not dropped. However, in LiDAR, besides the latency bound, it indicates the minimum required throughput to transfer the data. For camera streaming, Data Delivery Guarantee shows the minimum required throughput to function properly. The camera traffic is a video stream with low criticality, giving information for the positioning of the tool. The LiDAR sensor sends continuous data triggering an event-based reaction in case of an identified dangerous situation. The cloud-based PLC sends cyclic data to the robot, controlling the movement and behavior of the system. Since mobile robots need to collaborate with other robots or machines, a shared time zone is required, and synchronization between the end devices is needed. Therefore, the traffic types and requirements can be defined as shown in Table 1. Generally, the cyclic PLC data need to be transmitted in a shorter time than the latency bound. The robot is not generally sensitive to jitter, or packet delay variations, as long as the latency bound is kept. However, if two mobile robots are collaborating, the jitter in the transmission can lead to some varying offsets in the movements of the two individual robots. Depending on the required synchronicity in movement of the robots, this may require an upper bound on the tolerable jitter. Alternatively, the control of the two robots can be made with a common time reference that is shared among the robots, e.g., by time synchronization over the network.

3. Prototype of 5G Integrated with TSN

The 5G and TSN prototype development is divided into three parts: the wired communication using TSN, the wireless communication using a 5G URLLC pre-commercial test system, and the connection to end devices, including the mobile robotic platform and the factory cloud system.

3.1. Setup of the TSN Network

The TSN network consists of the endpoints receiving and sending messages as well as the TSN switches transmitting the data.

Setup of the TSN-Endpoints

For the integrated prototype, two different endpoints are needed: one sending the data and one receiving it. The TSN capability of the endpoints depends mainly on two factors: the correct and accurate sending of the data and the time synchronization to the rest of the system. To generate transferable results of the prototype, we decided not to use the robot PLC and the factory cloud as endpoints. For the prototype, two PCs were used that were both implemented in the same way, generating the same data traffic as the robot PLC and the factory cloud. In Figure 2, the PC endpoints and the potential connection points for the PLC and the factory cloud are indicated. To enable real time of the operating system, the kernel needs to be patched with a Preempt-RT Patch. With that patch, the Linux–Kernel is fully preemptible; therefore, it can be used as a real-time operating system [23]. Furthermore, to enable deterministic transmission and time synchronization, a network card capable of Precision Time Protocol (PTP) time synchronization and with at least two different transmission queues, one for time-critical and one for non-time-critical messages, is needed. Therefore, the Intel i210 network card was used, offering four transmission queues and hardware timestamping, which is needed for time synchronization using IEEE 802.1AS generalized Precision Time Protocol (gPTP).

Setup of the TSN Switches

As a set of standards, TSN offers a variety of mechanisms for reliable communication, most relying on shared time between all devices. To meet the requirements of the mobile robotics use case, deterministic data transfer needs to be enabled. This can be achieved by using the different traffic shaping mechanisms of TSN. Since TSN-enabled switches are not yet established in the market, the number of options and supported features was limited. The IEEE 802.1Qbv-Enhancements for Scheduled Traffic [24] enables shaping of the traffic in a time-driven fashion by controlling the egress gates of a switch for different traffic classes, thereby, e.g., enabling cyclic transmission of data. For instance, the transmission can be divided into cycles. Each cycle contains different slots in which only an assigned and specific type of data traffic is allowed. In addition to the time-synchronization data, two different data types are defined in the prototype: Best Effort Traffic and Critical Data (Priority Code Point 7). Corresponding to that, the cycles have been divided into slots, as shown in Figure 3. In classical best effort use cases, the traffic arrivals can be modeled with stochastic arrivals and ON–OFF distributions [25,26], and accordingly, scheduling algorithms are designed. For mission-critical industrial automation application use cases, traffic is typically periodic [15] and has been the focus in our work.

Deterministic delivery of data can only be achieved with IEEE 802.1Qbv time scheduling if all integrated components are time synchronized. Synchronization via Ethernet is offered by the IEEE 802.1 AS, gPTP [27] or IEEE 1588 Precision Time Protocol [28]. For the TSN network, two industrial switches with FPGA-based Ethernet ports (1 Gbps) supporting IEEE 1588, IEEE 802.1 AS and IEEE 802.1Qbv were installed with the two endpoints. This can be seen in Figure 2. With TSN switches synchronized to a common time, the IEEE 802.1Qbv time schedules of the different TSN switches can be configured in a coordinated manner so that prioritized traffic can be transmitted with deterministic performance through the TSN network. For the planning of the time schedules of the TSN switches, the minimum and maximum latencies of every TSN switch need to be considered. If large jitter is introduced by nodes in the network, the calculation of feasible time schedules for the TSN switches becomes difficult and even infeasible if the jitter becomes too large. Therefore, it is important to maintain low jitter throughout the network for applying TSN time scheduling according to IEEE 802.1Qbv.

3.2. Setup of the 5G-TSN-Bridge

3.2.1. URLLC Testbed

The standardization body for cellular communications, 3GPP, has put a high emphasis on enabling ultra-reliable low latency communication (URLLC) from the very beginning of 5G standardization. The 5G wireless technology standard is the only one that has been designed to target the requirements of a broad category of use cases requiring high reliability with real-time communication. The 5G technology aims at enabling URLLC with scalable deployments in industrial automation. The 3GPP body has specified several technical features to enable a high degree of reliability and low latency communication in Release specifications 15 and 16 on Industrial IoT [12]. The commercial availability of 5G products for Industrial IoT with TSN is still limited in the market. We use a pre-commercial 5G URLLC prototype with a standalone core network (cf. Figure 4) developed by Ericsson; it implements 5G time-critical communication features such as scheduling and feedback enhancements, robust control and data channels design, quality of service differentiation, prioritization handling, and efficient handling of the Ethernet traffic. In the URLLC system, the underlying algorithms and protocols are configured in a manner to satisfy real-time communication requirements.

The 5G URLLC system operates at 28 GHz (5G N261 band) and uses 120 kHz sub-carrier spacing. The 5G URLLC is a pre-commercial prototype and standard compliant standalone system with an on-premises core network [29]. Our empirical results in Figure 5 show that for over-the-air transmissions in a production environment of an industrial shopfloor with strong multipath propagation delays, the URLLC system is able to achieve reliable low-latency communication. The latency distribution on the URLLC testbed is shown in Figure 6 for a large sample set of 100,000 packets transmitted with traffic profile as listed in Table 1. It can be observed that although the latency is bounded to 1 ms for 99.9% value, there are very few outliers that go up to ca. 2 ms. One may observe that there is a spread in the latency distribution ranging from ca. 0.5 to 1.2 ms. The variations in the end-to-end latency in the URLLC testbed with over-the-air transmissions are caused due to several reasons. One of the reasons is the alignment delay between the packet reception in the URLLC testbed and the actual transmit opportunity in a TDD (time division duplex) system, where the same transmission spectrum bandwidth is alternatingly used for uplink and downlink transmissions. Moreover, the error control mechanisms impart an extra latency where an unsuccessful packet decoding at the destination is rectified through inherent HARQ (Hybrid automatic repeat request) and RLC (radio link control) retransmissions in the protocol stack. Finally, there are variations in the software/hardware packet processing delays both at the UE side and the network side.

3.2.2. Time Synchronization

The integration of the 5G-URLLC system in the TSN based network is shown in Figure 7. The URLLC system acts as a 5G virtual bridge in the TSN setup with the device and the network side TSN translator functions at the two endpoints [15]. The system clock of the network side translator function and base station are synchronized to the 5G grand master clock. The network side translator function performs the ingress/egress timestamping using the TSN clock. Inside the 5G URLLC testbed, the base station uses periodic System Information Block 9 (SIB9) messages as part of the Radio Resource Control (RRC) signaling to carry the timing information for the UE [30]. The 5G UE extracts the timing information from the SIB9 message to be used by the device side translator function to perform ingress/egress timestamping using the TSN clock. The periodic SIB9 messages allow correcting any errors accumulated during the periodic interval. A finer SIB9 granularity allows lower time synchronization errors and vice versa. In this article, we report results based on an SIB9 periodicity of 5 s.

3.3. Measurement Setup for Time Synchronization

For a precise and deterministic transmission of the messages, and for reliable measurements of the prototype performance, the synchronization between the different components is important, especially for the over-the-air performance validation. Therefore, a Pulse-Per-Second (PPS) signal is being used to compare the definition of time of each device. Using PPS signals, an offset between two or more clocks can be measured. For that, a Grand-Master Clock, with a PPS interface, is introduced to the system as a synchronization source. Furthermore, the TSN switch on the User-Equipment side of the 5G system is extended by a PPS interface. Connecting the PPS interfaces of the switch and the Grand-Master Clock to an FPGA (Field Programmable Gate Array) based PPS Analyzer from NetTimeLogic allows us to measure the over-the-air time synchronization with a resolution of 4 ns. The overall hardware architecture with the PPS measurement setup is shown in Figure 8.

3.4. Overall Architecture and Traffic Shaping

To meet the requirements defined in Section 2, the jitter, the latency of the transmission and the time synchronization of the communication need to suit the use case. Despite the high performance of the 5G URLLC testbed (cf. Section 3.2.1), the wireless communication introduces a jitter that does not suit the use case of mobile robotics. The same applies for the preempted Linux PC, which can prioritize the sender application but still introduces an undesired high jitter. Therefore, shaping of the traffic needs to be carried out using the TSN features, especially IEEE 802.1Qbv for jitter mitigation, as mentioned in Section 3.1. The traffic shaping needs to be carried out for the PC output and for over-the-air transmissions.

To use scheduled traffic in the TSN-based network, a precise arrival time is important to meet the defined cycles. Figure 9 shows the relevant communication aspects. The allowed arrival time window for critical data at the next TSN bridge on the communication path is the period in which the critical data need to arrive at the next bridge to meet the scheduled high-priority transmission slot. Data packets outside the time window miss the cycle and are retained until the next cycle, adding a whole cycle time to the latency of the message. To meet the time window, two aspects need to be considered: the inaccuracy of the time synchronization between two consecutive bridges and the overall transmission jitter, as shown in Figure 10. When the time synchronization is off, the messages will be sent at the wrong time and arrive outside of the allowed arrival time window. If the jitter is too high, the numbers of outliers increase, i.e., messages missing the scheduled time-window.

The scheduling of the TSN switches needs to be adapted to the performance of the end devices and the requirements from the application. As shown in Figure 10, TSN bridge 1 is able to reduce the jitter introduced by the TSN sender. By using IEEE 802.1Qbv, the gate control at the egress port is configured to hold the packets from the TSN sender for a certain time and then release them in a narrow time window. Depending on the performance of the sender and the bridge, this can be optimized either manually or by using a Centralized Network Configuration [7]. In the 5G prototype system, IEEE 802.1Qbv time scheduling is not implemented. Instead, the mechanism can be used at TSN bridge 2 to reduce the jitter introduced by the 5G system. Again, the messages transmitted through the 5G testbed are held back until the egress gate on the bridge is scheduled for priority traffic. By shaping the traffic using the scheduled traffic mechanisms of TSN, the jitter of the transmission can be reduced to a minimum. Furthermore, the jitter and the delay of each device in the transmission pipeline are not added to the overall pipeline jitter but can be eliminated at every TSN bridge in the network, thereby resetting the overall pipeline jitter. The various steps involved in traffic shaping in the integrated 5G and TSN setup are shown in Figure 10.

4. Discussion and Results

We have evaluated the performance of the integrating 5G with TSN in terms of the time synchronization accuracy, overall end-to-end latency and jitter behavior. All the measurements have been conducted in an industrial shopfloor with over-the-air transmissions. For validation, the data traffic with the highest requirements has been tested with 50 bytes message sizes and a cycle time of 2 ms, simulating the control data traffic in the mobile robotics use case described in Section 2.

4.1. Time Synchronization Accuracy

There should be a common time reference for field devices and the communication endpoints in the network. Time synchronization is a fundamental requirement for many of the TSN protocols. Reliable and accurate communication in industrial use cases require accurate time synchronization. As described in Section 3, we have used the NetTimeLogic PPS analyzer for precise time synchronization accuracy measurements of the prototype 5G-TSN integrated setup. The onboard FPGA on the PPS analyzer computes the time difference between the PPS signals at the two endpoints of the setup of 5G integrated with TSN, as shown in Figure 8 with nanosecond accuracy. We have carried out PPS measurements for the time synchronization error of the 5G system integrated with TSN for over 6 h. As shown in the histogram in Figure 11, the mean time synchronization accuracy in the setup of 5G integrated with TSN remains below 3 µs, while the maximum error remains below 8 µs. This level of time synchronization accuracy in the integrated setup (including over-the-air time synchronization in the URLLC test system) is sufficiently well suited for the IEEE 802.1Qbv based gating scheme implemented in our prototype setup for de-jittering purposes, as described in the next section.

4.2. Dejittering of the Transmission

In Section 4, the scheduled traffic and traffic shaping for the integrated 5G and TSN communication pipeline has been shown, especially the shaping of the PC output and the shaping of the over-the-air transmission time. To validate the shaping of the output, measurements with cyclic data traffic and a fixed message size have been performed. A message size of 50 bytes and a cycle time of 2 ms have been tested with at least one hundred thousand samples. To determine the performance of the 5G-TSN bridge independent of the PC performance, the jitter has been measured at the output of the PC sending the data, the egress gate of TSN-Switch 1 and the egress gate of TSN-Switch 2. The performance of the URLLC testbed can be found in Section 3.2.1.

The shaping of the PC output can be seen in Figure 12. On the left side, the jitter of the PC output is shown, and on the right side, the jitter at the egress gate of TSN-Switch 1 is shown. Since the PC is not a dedicated real-time system, the computation of the application and the cyclic sending of the data generates a certain output jitter that can go up to 50 µs and with a 99.9% value at ≈21 µs. Using the TSN-Switch 1 to reduce the jitter by holding the messages for a certain time and allowing the PC to send within a certain time window, the probability distribution of the messages is reduced to a 99.9% value of 125 ns, as shown on the right side of Figure 12. This is a significant reduction in the jitter caused by the PC, thereby enabling deterministic communication with narrow time arrival windows. Especially for fieldbus systems or scheduled traffic in industrial Ethernet networks, the time windows are short to enable fast communication. Our empirical results signify that without requiring expensive real-time hardware, factory cloud and edge systems could satisfy real-time application requirements with appropriate traffic shaping.

In Figure 13, the measurement results of the integrated 5G and TSN communication pipeline without IEEE 802.1Qbv are shown. On the left side, the output jitter of the PC is shown with a 99.9% value of ≈35 µs, and on the right side, the jitter measured at TSN-Switch 2 is shown with a 99.9% value of ≈373 µs. Considering the latency values of the 5G-URLLC-testbed presented in Section 3.2.1, it can be seen that the jitters of the PC and the URLLC-testbed are accumulated in the pipeline. The overall end-to-end jitter is increasing with each node in the communication chain. As a consequence, in a large-scale network without traffic shaping mechanisms to reduce the jitter, such as traffic scheduling according to IEEE 802.1Qbv, significant jitter can accumulate over the network and make it possibly unsuitable for certain jitter-sensitive industrial usage.

In Figure 14, the measurement results of the integrated 5G and TSN communication pipeline with IEEE 802.1Qbv scheduled traffic are shown. Again, the PC is introducing a jitter with a 99.9% value of 16 µs but the egress jitter of TSN-Switch 2 is reduced to a 99.9% value of ≈920 ns. With the reduction in jitter, it can be seen that not only the jitter introduced by the PC but also the jitter introduced by the 5G system is compensated by the traffic scheduling mechanism. Using IEEE 802.1Qbv, the wireless 5G bridge can be integrated in the network in a way that the value of the transmission jitter is similar as in a wired connection; see Figure 12. By decoupling components introducing larger jitter with TSN switches, the jitter can be significantly reduced, making 5G suitable for jitter-sensitive deterministic communication for industrial usage. Compared to Figure 13 with the 99.9% value of the jitter being at 372 µs, the 99.9% jitter over the integrated 5G and TSN network can be reduced by a factor of 400 to only 922 ns by applying IEEE 802.1Qbv, as shown in Figure 14. These results clearly show the potential of 5G integrated with TSN for the industrial usage.

4.3. Evaluation of the Implementation Regarding the Use Case

In Section 2, we discussed the motivation behind the use case and utilizing 5G in combination with TSN for cooperative tasks between mobile robots. As mentioned before, the reliable real-time communication between the factory cloud system and the mobile robots is vital to enable the synchronized movements of mobile robots. The factory cloud system calculates and transmits the Robot Operating System (ROS) based commands related to the path planning, manipulator movement, obstacle avoidance, and advanced AI/ML algorithms for object detection to the mobile robots. The factory cloud system transmits control commands to the mobile robots periodically with the cycle time of 7 ms with the packet size ranging from 32 to 80 bytes. Since control commands are critical, the PLC is configured such that the data should arrive at the receiver side within a latency bound of 6.3 ms (90% of 7 ms cycle time). Otherwise, the mobile robot will come to a halt, since the controller raises a critical error due to the safety requirement. Hence, communication between the factory cloud system and the mobile robots—including the TSN network and the 5G system—needs to fulfill this upper latency bound. As measurement tests indicate, the 5G system integrated with the TSN network provides a latency below 0.8 ms with high reliability. Such reliability ensures that the critical traffic is transmitted within the required time window for the use case.

In our use case, mobile robots transmit their LiDAR data and camera data to the factory cloud system. LiDAR data are classified as a medium traffic. LiDAR data periodically transmit data to the factory cloud system every 25 to 100 ms (which is configurable) in order to provide essential data for obstacle avoidance and emergency stop. LiDAR data are part of the uplink communication in our 5G and TSN setup with the required throughput of around 1 to 3 Mbps. LiDAR data are transmitted to the factory cloud system in real time with high reliability within the latency below 1 ms. On the other hand, camera traffic is considered as background traffic with large packet sizes, which requires higher throughput compared to LiDAR but without strict latency requirement on the wireless communication. The challenge for the camera data is the high throughput, which is limited in the uplink direction by the uplink capacity of around 150 Mbps. In the use case, the resolution of the camera is reduced to keep its throughput within 20 to 50 Mpbs while not yet degrading the performance of AI/ML-based object detection running on the factory cloud system. Based on the measurement test for the data packet size of 1042 bytes, the maximum latency for over-the-air communication stays around 1 ms, which is much lower than what is required for data transmission of the camera.

The coordination of mobile robots demands 5G and TSN to fulfill the traffic requirement of both mobile robots at the same time. In this case, the robot controller in the factory cloud system receives the necessary data from both mobile robots and schedules the movement of each of the mobile robots to execute the coordination tasks. The challenge of coordination tasks in our use case is syncing the movement of mobile robots specially when they handle sensitive materials such as glasses, since any jitter that causes the latency of communication between the controller and the mobile robot to go beyond the maximum latency bound causes one mobile robot to stop operating. In this case, the robot controller in the factory cloud needs to transmit a halt command to the other mobile robot within 7 to 10 ms; otherwise, the glass will break. The aforementioned time period includes any processing time of factory cloud system, which needs to be accounted for determining the permitted latency for the 5G system to ensure the safe operation of mobile robots in coordination tasks. During the measurement test period, we have not experienced any latency exceeding the maximum latency bound, and both mobile robots stayed in operating mode during the execution of the task during the experimentations.

5. Conclusions

In this paper, a prototype 5G system integrated with a TSN network has been presented for a typical industrial mobile robotics use case. The prototype consists of a 5G URLLC test system and commercially available TSN switches. Different TSN protocols were integrated with the 5G system for deterministic over-the-air communication. The paper describes the architectural details and step-by-step analysis on jitter reduction mechanisms applied in the communication chain between the application hosted in a factory cloud and the mobile robot. The selected use case of a mobile robot will play an increasing role in manufacturing in the future to increase production flexibility by, e.g., introducing line-less production systems. It puts high requirements on communication with regard to low latency, high reliability and low jitter. The experimental validation took place in an industrial shopfloor environment. In the context of 5G system integration with TSN, these are the first measurements taken in such an environment.

With the TSN standards used, the performance results of the prototype satisfies the requirements of the use case, enabling the wireless control of mobile robots. Time synchronization over the 5G system can be achieved at high precision; the time error introduced by synchronizing over the TSN network—and including the 5G system—was observed to be lower than 8 µs with a mean value of below 3 µs in our experimental validation over the test period of more than 6 h. On the industrial shopfloor, the 5G URLLC test system demonstrated time-critical communication performance; we observed an end-to-end latency over the TSN network and the 5G system that was below 0.8 ms with a 99.9% reliability. However, a jitter in the transmission over the 5G system in the order of 500 µs was experienced, which is due to the characteristics of the 5G design and the wireless characteristics. We studied an end-to-end TSN configuration using TSN time scheduling according to IEEE 802.1Qbv, where all network nodes are synchronized to a common reference clock. One key measurement result is that the jitter introduced by different entities in the communication path can be isolated and compensated in a TSN-based network, reducing the overall jitter end-to-end. In particular, by having an IEEE 802.1Qbv-configured TSN bridge located after a node with relatively large jitter, the IEEE 802.1Qbv-configured bridge can compress this jitter. In our experiments, we could reduce the end-to-end jitter over the TSN and 5G network from ≈370 to ≈0.9 µs through IEEE 802.1Qbv configuration in the network.

From our test setup and validation, we conclude that 5G, and in particular when integrated with an industrial TSN network, is capable of supporting demanding industrial automation use cases. We have implemented and validated this for the example of cloud-controlled mobile robotics in an industrial shopfloor environment.