{'fraction_non_alphanumeric': 0.03405975558756563, 'fraction_numerical': 0.008875052274522559, 'mean_word_length': 4.549767921609077, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reduction of Complexity in Q-Learning a Robot Control for an Assembly Cell by using Multiple Agents

Production systems in Industry 4.0 are characterized by a high degree of system networking and adaptability. They are often characterized by jointed-arm robots, which have a high degree of adaptation. Networking and adaptivity increase the flexibility of a system, but also the complexity of the control, which requires the use of new development methods. In this context, the Simulation-Based Control approach, a model-based design method, and the concept of Reinforcement Learning (RL) are introduced and it is shown how a task-based robot control can be learned and executed. Afterwards, the time complexity of the Q-learning method will be examined using the application example of a robot-based assembly cell with two differently flexible system configurations. It is shown that, depending on the system configuration, the time complexity of learning can be significantly reduced when using several agents. In the studied case, the complexity decreases from exponential to linear. The modified RL structure is discussed in detail.


Introduction
Production systems of Industry 4.0 have a high degree of networking and adaptivity. The latter characterizes the flexibility of a system to adapt to changing influences [1]. Systems are often characterized by jointed-arm robots which, according to [2], have a high degree of flexibility in terms of design and control. Adaptivity and networking increase the complexity of the control software, which requires the application of new development methods. In recent years, similar methods have been established under various terms, such as Model-Based Design (MBD) [3], Rapid Control Prototyping (RCP) [4] or Virtual Commissioning (VC) [5]. What they have in common is that they are based on a continuous model-and simulation-based development from the design to the operation phase. The Simulation-Based-Control (SBC) approach [6] was developed adequately for this purpose by the Computational Engineering and Automation research group at the University of Applied Sciences in Wismar. This approach was adapted in [7] and [8] specifically for task-oriented control development for jointed-arm robots. In [9], a connection of the SBC approach with machine learning methods based on Reinforcement Learning (RL) according to [10] is shown. The RL is defined by a structure of at least one agent, which has a learning method and an environment. The specific learning method is Q-learning. Based on predefined tasks and transformation modules, a control strategy is learned and automatically transformed into an SBC-compliant program. Since Q-learning is a model-free algorithm, it can be applied to various problems. However, impracticable computing times result relatively quickly, even though learning can often be accelerated for problems with limited state space by means of parallel processing and binary trees [11]. A significant reduction of the state space and, thus, the computational effort is achieved with model-based RL approaches [12]. Here the agent already has process-relevant knowledge at the beginning of the learning process, but thereby loses its universality. For problems with large state space, the computing time can be reduced by combining artificial neural networks (ANN) as function approximators [13], [14]. However, this increases the complexity of the software architecture. Furthermore, the design and training of the ANN requires experience and time. In contrast, the simple architecture of the original Q-learning is an advantage if the computational effort can be mastered.
In this paper, we investigate how the computing time for Q-learning of a task-based robot control can be reduced by using several agents. Two differently flexible system structures of an assembly cell are considered and the time complexity of learning with one agent compared to several agents is analyzed. The learning is performed on simulated system environments. The generation of an executable robot control based on the SBC approach and basics of the RL approach using Q-learning are discussed in the following background section.

Background
Starting from the adapted SBC framework for articulated arm robot systems according to [7] and [8], this section deals with the principle of integration with a machine learning process and the basics of RL based on Q-learning according to [10] and [13].

The SBC Approach
As shown in Figure 1, an SBC-based robot control is layered in analogy to the concept of the Robot Operating System (ROS) [15]. The Control Model (CM) specifies the control strategy by composing predefined basic tasks. The Process Model (PM) implements the task transformer based on predefined task-specific modules. Additionally, the PM maps the state information. The Interface Model (IM) implements the interface to the hardware using a robot middleware. With the RCV Toolbox for MATLAB according to [16] as middleware, SBC-based robot controls can be developed and operated independently of robot manufacturers and model-based with MATLAB/Simulink. Virtual and real robots can additionally interact with a virtual process environment in the form of a simulation model.
With the integration of the SBC framework with a RL procedure, as introduced in [9], the task-based control specification of the CM according to formula 2 is automatically generated from a sequence of learned state/action tuples according to formula 1: Learning takes place offline using a simulated process environment. In principle, the learning algorithm could also run during the operating phase and adapt the control strategy in certain time windows if the real-time requirements are met.

Reinforcement Learning with Q-learning method
Besides supervised and unsupervised learning, the RL method forms a third class of machine learning procedures. The goal of RL is to learn a behavioral strategy : → that assigns an action ∈ to each state ∈ . The RL does not require explicit training data. It trains itself using a real or simulated environment according to the trial and error principle. The RL is based on a framework as shown in Figure 2. In model-free RL, the agent only knows the allowed action set at the start of training. The environment is defined by different states, ∈ . When an action ∈ takes effect, the environment determines a subsequent state ′ with the state transition model : → and computes a reward value ∈ for the current action. The subsequent state ′ and the reward are sent back to the agent. During training, the agent receives information about the possible states of the environment and the benefits of actions through iterative interaction, and gradually learns a behavioral strategy . After completion of all training episodes, the behavioral strategy is derived from the Q-matrix. Learning takes place in phases, also known as episodes. These are independent of each other, always start in an initial state � of the environment and end when a target state ������ or abort state ����� is reached. At the beginning of the training, the agent selects an action ∈ purely randomly (exploration). As the learning process progresses, the agent increasingly uses the knowledge it has acquired to select an action (exploitation). The ratio of exploration to exploitation is adjusted in the course of training.
Q-learning is based on a table function called Q-matrix. A matrix element ( , ) represents the benefit of an action when it is performed in the state of the environment. From the cardinality | | follows the column dimension of . The row dimension of grows dynamically during the training with the number of states explored. The data of explored states are stored in an indexed data structure based on the row index of . This allows the agent to clearly recognize already explored states. The first episode of the training starts with an empty Q-matrix and an empty data structure. Subsequent episodes build on the already acquired knowledge in and . After each interaction with the environment, the agent checks whether the received state ′ is known. If not, the Q-matrix is extended by a new row and the state data are added to the data structure . The successive adjustment of the Q-values results from formula 3 according to [13]: The updated Q-value of the current state/action tuple ( , ) is calculated from the previous Q-value, the currently received reward , and the maximum Q-value of all possible actions in the currently received subsequent state ′. The influence of the individual variables is determined by the hyperparameters: (i) discount factor and (ii) learning rate . The discount factor controls the influence of rewards expected in the future and the learning rate controls the influence of the current observation. In environments with deterministic behavior, a high learning rate can be applied with up to = 1.

Application Example
An automated assembly cell (AC) with an articulated arm robot for the production of different assemblies is considered as an example. By means of RL, product-compliant assembly sequences are to be learned, on the basis of which robot controls can be generated according to Section 1.1. Figure 3 shows the system layout of the AC. The AC consists of the articulated arm robot (R), an assembly station (AS) and a belt conveyor (BC). The BC feeds input parts from upstream production sections and serves as a transfer zone (TZ) to the robot. Depending on the specific system configuration, different numbers of transfer locations (TL) are possible, which influences the system flexibility. Unused input parts and assembled modules are automatically removed. The sequence in which the individual parts enter the AC is unknown and is assumed to be random. Figure 4 shows an example of an assembled module as an exploded view, whose assembly is examined below for two system variants of the AC: 1. Minimum variant with only one TL as TZ and 2. Variant with three TLs as TZ.
For the desired complexity consideration, the assembly is reduced to a pick & place application. The assembly of the module in Figure 4 is subject to six rules. Four of the rules are shown in [7]. Rules 5 and 6 were added: 1. Base plate (1) must be mounted first. 2. Pins (4, 5) may only be mounted after cover (7). 3. Comprehensive parts must be mounted after enclosed parts. 4. The glass cover (8) must be mounted last. 5. Each part is only assembled once and the pins (4,5) are not identical. 6. Gripping attempts on empty TLs of the TZ are prohibited.
In the following two sections, the learning of an assembly strategy with first one agent for both system variants and then using several agents for the second variant is examined.

Learning with one Agent
According to Section 1.2, the RL framework consists of an agent and an environment. In the application case under study, the robot forms the agent and the BC and AS form the environment. In this section, the learning of an assembly strategy based on this RL framework is examined for the two system variants of the AC. A state ∈ of the entire environment is, therefore, described by a 9-tuple. The first eight elements describe the assembly state on the AS and the ninth element the state of the TZ. Figure 5 shows the representation of states ∈ of the environment with a state vector (SV). State � represents the initial state, � a possible later subsequent state and ������ the target state. The indicated state � encodes an assembly of the components' base plate and labyrinth as well as the allocation of the TZ with a component glass lid. The target state ������ is reached when the first eight elements of are not equal to zero. The character in the initial and target state stands for any state �� ∈ �� of the TZ.

System variant with one TL as TZ
The environment reacts to an action of the agent, as shown in Figure 2, with a reward value and a subsequent state � . The reward model of the environment defines three possible rewards for an action = 1 based on the six assembly rules according to Section 2: With a permitted action = 1 (Pick&Place), the last element of the SV that represents the part in the TC is restored according to its value in the SV (lines 4-7), thereby updating the assembly state. In addition, for each allowed action = 1 or = 0 the last element of SV is assigned a random integer value from the interval [0, _ ], which represents a new input part in the TC (line 8).
Each episode starts with the transfer of a start state � of the environment to the agent. A reward is not transferred at the beginning of an episode. Learning is done as described in Section 1.2. The column dimension of the Q-matrix corresponds to the cardinality of the action set |A| = 2. An episode ends when the agent receives a reward, = 1, from the environment. The training must include enough episodes to learn a strategy, . It should be noted that, with the random generation of new input parts in the TZ, the state transition model is subject to stochastic influences.
Without assembly rules, this system structure results in 2 � ( + 1) − states of the environment depending on the number of parts to be assembled. Due to assembly rules, the number of states is considerably reduced. For the assembly under consideration with eight components, 144 states result according to [7].
The complexity of an algorithm as a function of the input data is described with the big notation [17]. An empirical analysis showed that a linear time complexity ( ) of the learning process resulted depending on the number of components, , to be assembled. The computing time for learning an assembly strategy was less than one hour on a standard PC with an implementation in MATLAB.

System variant with three TLs as TZ
In the variant with three TLs the allocation of the TZ is analogous to a shift register. In fixed time units, the BC moves one position to the right. The robot always has access to all three TLs. This results in four possible actions for the robot, A = {0,1,2,3}. The action a = 0 again encodes the task None and the other three actions encode the task Pick&Place with different parameterization depending on the TL to be approached.
The state of the environment expands by two elements to an 11-tuple. The first eight elements describe the assembly state on the AS. The other three elements each describe the non-occupancy or occupancy of a TL. Due to the three TLs, the number of possible states of the environment increases. For the TZ with three TLs and eight possible input parts, as well as the case of non-occupancy of a TL, results in 729 states �� ∈ �� with �� = {(0,0,0), (1,0,0), … , (5,1,1)}. The values 1 to 8 encode a component according to the part numbers in Figure 4 and the value 0 encodes the non-occupancy of a TL.
The calculation of rewards r ∈ is analogous to the variant with only one TL. The state transition model , which has been extended to a TZ with three TLs, shows Listing 2.
The 11-tuple representing the state of the environment is again implemented as a state vector (SV). If the agent selects a permitted action a = {1,2,3} for a Pick&Place task (line 2), the input part is taken from the corresponding TL in the TZ and mounted at the AS by the robot (line 4-7). Subsequently, for each allowed action a ∈ {1,2,3} of the robot, the shift register movement is realized by the BC (lines 8-11) and a new input part is randomly generated in the form of an integer value in the interval [0, _ ] for the first TL in the TZ (lines 12-13).
Learning a behavioral strategy is analogous to the representation in Section 3.1. However, despite the applicable assembly rules, the state space of the environment increases exponentially with 144⋅9⋅9=11644 states. The complexity of the learning algorithm corresponds to ( � ) with the number of components to be assembled and the number of TLs in the TZ. In an empirical study, the computing time was about 10 hours and it was about ten times longer than the computing time for the variant with only one TL.

Learning with Multiple Agents
Based on the structure of the application problem, a multi-agent approach to reduce the computing time appears to be appropriate for the second system variant with three TLs in the TZ. Figure 6 shows an RL framework consisting of three agents, an environment, and a management component.
The three agents are identical. Their structure and behavior correspond to the individual agent in Section 3.1, and their action set = {0,1} represents the two tasks None and Pick&Place. The models and of the environment correspond to the representation in Section 3.2. Accordingly, the environment reacts to the action set = {0,1,2,3}. In contrast to the original RL framework, the agents and the environment do not communicate directly with each other, but via the intermediate management. The management decomposes each state � as shown in Figure 7. The first eight elements of � , which encode the assembly state on the AS, are passed on to all three agents. From the ninth to the tenth element, which  code the allocation of the three TLs in the TC, each agent only receives the state of one TL at a time. This reduces the state space from the viewpoint of each individual agent to 144 states, analogous to the agent in Section 3.1. An episode begins with the decomposition of the initial state � of the environment by the management, analogous to Figure 7. Then, the management sends to each agent the corresponding reduced state vector (�) ′, as shown in Figure 6. Each agent reacts individually with an action (�) ∈ {0,1}. The management selects one of the three actions according to the following criteria: 1. An action = 1 (Pick&Place) is preferred over an action = 0 (None).
2. If several agents send an action = 1, the agent with the highest index is selected. The prioritization of the agents follows from the way the TZ works as a shift register. Parts on the second and first TL are still available in the next and next but one cycle. According to Figure 7, 1 has the lowest priority and 3 the highest.
The management remembers the selected agent. If an action = 0 was selected, the action is forwarded to the environment unchanged. If an action = 1 was selected, the action is converted into a value of the action set = {1,2,3} based on the index of the agent and sent to the environment as action . The environment reacts to the action with a reward and a follow-up state ′. These are calculated with the reward model as well as the state transition model according to Section 3.2. The management sends the reward as (�) to the agent , whose action was selected before. The other agents receive an empty reward value. An empty reward means that there is no learning for the agent in this iteration step because its action has been discarded. The subsequent state � is decomposed by management as described and the reduced states (�) ′ are sent to the agents. Each agent learns a behavior strategy (�) related to the TL assigned to it. Finally, an overall assembly strategy is derived from the individual strategies.
The number of necessary episodes is hardly different from learning with one agent. Since only one agent is active in each iteration step, the state space to be analyzed has a maximum of 144 states. The empirical investigations revealed a linear complexity ( ) of learning depending on the number of components to be assembled. The multi-agent approach is scalable regarding the selection of components in the TZ and the complexity is, therefore, independent of the number of TLs. The computing time requirement is reduced to about one hour.

Conclusion and Outlook
Starting from the basics of RL using the method of Qlearning and the model-based SBC approach, it was shown how, in principle, a task-based control can be learned and executed.
Subsequently, the learning of a typical pick and place strategy for two differently flexible system structures was considered using the example of a robot-based assembly cell. The focus of the consideration was the time complexity of learning. For the simple system structure with binary selection option, it was found that the learning algorithm has a linear complexity ( ) depending on the number of assembled components. In contrast, the second system structure with many simultaneous choices per assembly step had an exponential complexity ( � ) of learning.
As a result, an RL multi-agent framework with the Qlearning method was designed for the second system structure. It could be shown that, for learning with the multi-agent approach, a linear complexity ( ) results as a function of the number of components to be assembled and that the complexity is independent of the number of simultaneous choices due to the scalability of the approach.
The application example under study is characterized by learning an assembly sequence that conforms to the assembly. The different system structures were purposefully modelled as environments for the RL according to the MDP paradigm. In subsequent investigations, further influencing variables of a production process are to be taken into account during learning, such as the introduction of input parts as a function of buffer capacities. This requires the integration of the RL with a more complex dynamic simulation model of the production, which was not explicitly developed according to the MDP paradigm. Conceptual approaches for such simulation-based RL experiments are presented in Schmidt [18] and Adams [19].