{'fraction_non_alphanumeric': 0.04258196571512117, 'fraction_numerical': 0.014985927848240068, 'mean_word_length': 4.4167491585824585, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unentangled quantum reinforcement learning agents in the OpenAI Gym

Classical reinforcement learning (RL) has generated excellent results in different regions; however, its sample inefficiency remains a critical issue. In this paper, we provide concrete numerical evidence that the sample efficiency (the speed of convergence) of quantum RL could be better than that of classical RL, and for achieving comparable learning performance, quantum RL could use much (at least one order of magnitude) fewer trainable parameters than classical RL. Specifically, we employ the popular benchmarking environments of RL in the OpenAI Gym, and show that our quantum RL agent converges faster than classical fully-connected neural networks (FCNs) in the tasks of CartPole and Acrobot under the same optimization process. We also successfully train the first quantum RL agent that can complete the task of LunarLander in the OpenAI Gym. Our quantum RL agent only requires a single-qubit-based variational quantum circuit without entangling gates, followed by a classical neural network (NN) to post-process the measurement output. Finally, we could accomplish the aforementioned tasks on the real IBM quantum machines. To the best of our knowledge, none of the earlier quantum RL agents could do that.


I. INTRODUCTION
Classical reinforcement learning (RL) [1] has generated excellent results in different regions [2][3][4][5][6][7]. During the past decade, RL has been broadly applied to master Go [2], design chips [7], play the game for StarCraft and Gran Turismo [3,4], improve the nuclear fusion problem [5], and solve the problem of protein folding [6]. Despite the remarkable achievements, most RL techniques fail to balance the tradeoff between exploitation and exploration [8]. The difficulty comes from the fact that the state-action space is exponentially large [9], where the optimal policy can not be explored efficiently. A mainstream strategy is to feed numerous trials to RL models in the optimization process to enhance performance [10][11][12][13][14][15][16][17]. Nevertheless, such a sample inefficiency challenges the applicability of RL towards large-scale problems [8], where the requested computational overhead is expensive or even unaffordable. For example, in the task of playing an Atari game, two representative RL models, i.e., Deep Q-learning [18] and Rainbow RL [19], achieve good performance after about 80 and 300 hours of play experience in an Atari game, while humans learn it within a few minutes. In this regard, improving sample efficiency (the speed of convergence) is the key to using RL to solve complex real-world problems.
Besides the attempts to understand the potential advantages of quantum supervised learning models, there is a growing interest in designing powerful QRL models to compensate for the caveats of classical RL models, such as sample inefficiency. There are not proven theoretic results regarding the advantages of QRL models up to date. Instead, most studies numerically evaluate the performance of their proposals [67][68][69][70][71]. Concretely, Refs. [46][47][48][49][50][51][52] have attempted to improve sample inefficiency by using multi-qubit variational quantum circuit (MVQC) [72,73] that has lots of entangling gates on a generic benchmark OpenAI Gym [74]. However, none of them outperforms classical RL models [47][48][49]52]. Alter-  [76][77][78] to improve the sample efficiency. Unfortunately, these algorithms are hard to implement on NISQ devices. Considering the ambitious aim of QRL is to provide computational advantages over classical NN on real-world tasks, it is natural to ask: "Do the current QRL agents surpass the classical RL agent in OpenAI Gym? " If the response is positive, it is necessary to figure out "how would we design the model? "

A. Main results
We demonstrate a series of training and testing processes to address the previous question. We first propose a single-qubit-based variational quantum circuit (SVQC) model that only consists of single-qubit rotational gates. Our SVQC models show the better convergence compared to the classical fully-connected neural networks (FCNs) on the learning curves in the tasks of CartPole and Acrobot, and can use much (at least one order of magnitude) fewer trainable parameters than the classical FCNs to accomplish comparable or better learning performance. Furthermore, our SVQC models achieve higher rewards than other VQC-based models in the CartPole and Acrobot tasks [47][48][49]51]. While we first successfully train the quantum agent to accomplish the LunarLander task in the QRL field, our trained models also exhibit satisfactory performances in the testing tasks of CartPole-v0, Acrobat-v1 and LunarLander-v2 on the IBM quantum devices.

B. Related work
This section collects related works, where VQC-based quantum RL agents were used to solve tasks in the Ope-nAI Gym. For ease of comparison, these results are summarized in Table I. Specifically, the Frozen Lake envi-ronment in toy text tasks was first solved by Chen et al. [46]. The CartPole-v0 task was first attempted in Ref. [47] with quantum Q-learning, but its performance is not satisfactory. The control tasks of CartPole-v0, MountainCar, and Pendulum were subsequently accomplished in Ref. [48][49][50]. The employed learning algorithms in [46,[48][49][50] were also included in Table I. However, whether the VQC-based model can accomplish the more challenging tasks in OpenAI Gym, e.g., CartPole-v1, LunarLander-v2 and box2d, remains to be answered.
Finally, we remark that the aforementioned tasks were conducted using ideal simulators. It is unknown whether noisy quantum RL agents could achieve satisfactory performance.
The paper is organized as follows. Preliminary of classical RL and VQC-based QRL are described in Section II. A novel variational QRL with single-qubit is described in Section III. The simulation results and associated discussions are presented in Section IV. The concluding remarks and open questions are presented in Section V.

II. PRELIMINARY
Here we briefly recap classical reinforcement learning (RL) in Section II A, RL with variational quantum circuit (VQC) in Section II B and introduction to the OpenAI Gym in Section II C.

A. Classical reinforcement learning
Markov decision process (MDP) [79] provides a dynamical framework that captures two key features in classical reinforcement learning (RL); namely, trial and error as well as delayed rewards [1]. An MDP can be described as a 5-tuple, M = (S, A, P, R, γ), where S ∈ R d is the state space, d is the dimension of states, A ∈ R is the action space, P(x i+1 |x i , a i ) ∈ P is the probability of transitioning into state x i+1 upon taking action a i in state The environment provides the input state's features xi at step i. The features are encoded to the quantum state by parameterized circuits. Then, the quantum state evolves with the parameters layer U(θi) and the state is measured by projective operator repeatedly. The classical computer calculates the loss function by the scaled circuit output based on the measurements. The agent chooses an action ai through the policy, which is dependent on the circuit output, and the agent executes the action on the environment. It receives the reward and next state's features xi+1 from the environment. The classical computer calculates the new loss function dependent on the rewards and policy. The trainable parameters θi, a ik , and b ik are updated to minimize the loss function.
x i , (x i , a i ) ∈ R d , R is the state-action pairs at step i, R ∈ [0, R max ] is the reward space, R max ∈ N is a constant, and γ ∈ [0, 1) is the discount factor [80]. An agent begins at an initial state x 0 sampled from an initial distribution P(x 0 ). Then it implements the policy π(a i |x i ) ∈ [0, 1] to take action a i ∈ A at step i from a state x i ∈ S and moves to a next state x i+1 ∼ P(·|x i , a i ). The next state x i+1 is dependent on the current state x i and the agent's action a i . After each action, the agent receives a reward r i = R(x i , a i ) ∈ R. Therefore, the relation between action and reward is similar to trial and error, and the reward function r i is associated with delayed rewards.
The goal of modern RL with classical neural network (NN) is to maximize the discount expected rewards where τ = (x 0 , a 0 , x 1 , a 1 , . . . , x i , a i ) is the trajectory in an episode,Ê π [. . . ] denotes the expectation value under all possible policies. For the agent with the state x i at time i, the probability of the agent to take the action a i is π θi (a i |x i ). The agent learns the stochastic policy π θi (a i |x i ) that is dependent on x i and θ, where θ i ∈ R m is trainable parameters in classical NN, m ∈ N is the dimension of parameters to reach high expected reward. The policy is improved through updating the parameters by gradient ascent Defining the loss function plays a crucial part in optimization problems. The loss function of the PPO-clip is as follows: is the ratio of new and old policy, θ i is the policy parameters before the update, ∈ R is usually a small number. PPO-clip is an robust algorithm in various experimental tests [81]. Moreover, it is commonly used in RL algorithms for the OpenAI Gym because it is easily operated with good performance.

B. Reinforcement learning using variational quantum circuit
The VQC-based QRL is illustrated in Fig. 1. The key concept of VQC-based QRL [46][47][48][49][50][51][52] is to learn the policy to acquire the maximum expected rewards R i (τ ) = E π T t=0 γ i r i (x i , a i ) by replacing the classical NN with VQC.
First, the input data features x i at step i from the environment are transferred as |Ψ in (x i ) = U(x i )|0 , where |0 ∈ C 2 n is the initial state, n ∈ N is the number of qubit, |Ψ in (x i ) ∈ C 2 n is the quantum state after encoding the input x i , and the operator U(x i ) is an unitary dependent on x i . Then, the quantum state |Ψ in (x i ) evolves by the operator U(θ i ), where U(θ i ) is an unitary, θ i ∈ R m is the trainable parameters in VQC, m ∈ N is the dimension of parameters. The resultant quantum state is measured by projective operator. A general form of circuit output is where U(x i , θ i ) ∈ M 2 n is an unitary operator that depends on the input and trainable parameters, and M ∈ M 2 n is a projective operator. Second, the circuit output based on the measurement is scaled linearly by parameters, The agent's policy decides the probability of action a depending on the scaled output where a i is the action at the state x i , and k is the number of actions. After interacting with the environment, the agent receives the reward and next state x i+1 . The loss functions L(x i , θ i ) ∈ R are dependent on the scaled output and the cumulative rewards. Finally, all trainable parameters, namely θ i , a ip , and b ip are optimized by gradient descent on a classical optimizer.

C. OpenAI Gym environments
OpenAI Gym provides benchmarking environments for RL tasks to compare their model performance. CartPole, Acrobot, and LunarLander tasks are regarded as the basic environments in OpenAI Gym. The schematic diagrams and the performance metrics which measure how well RL agents can achieve the intended goals of these three tasks are shown in Fig. 4 (a)-(c) and Table II, respectively.
The goal of the CartPole task is to balance the pole on a cart by moving the cart. A reward of +1 is given for each step that the pole remains upright. The episode ends when the pole is more than 12 degrees from vertical, or the cart moves more than 2.4 units from the center. Acrobot-v1 is another classical control task in the Gym. The system includes two joints and two links, where the joint between the two links is actuated. Initially, the links are hanging downwards, and the goal is to swing the end of the lower link up to a given height.
LunarLander is a more complex task than CartPole and Acrobot tasks. Its goal is to let the agent learn to land between the two yellow flags. The agent controls up to four actions corresponding to no action, the main engine is firing down, and the engine is firing left or right.
More descriptions about the CartPole, Acrobot and LunarLander environments together with discussions on input data encoding schemes, measurements and VQCbased quantum RL can be found in Appendix A.

III. A NOVEL VARIATIONAL QUANTUM REINFORCEMENT LEARNING WITH SINGLE-QUBIT
To improve the performance based on the VQC method in OpenAI Gym tasks, we propose a new architecture, SVQC, which is composed of three parts: input, parameter, and output layers shown in Fig. 2. For input layer, the environment provides an input state x i to the input layer. The state x i is encoded by the angle of a singlequbit rotational gate U(x i ). The number of the data encoding rotational gates is usually determined by the state dimension.
In the parameters layer, trainable parameters θ i control the single-qubit rotational gates U(θ i ) and are updated through gradient descent. Here, we only use the single-qubit gates without entangling gates to overcome the problem of the barren plateau caused by entangling gates in the optimization process [82].
The output layer is decomposed by three components: The architecture composes of three parts: input, parameter, and output layers. For input layer, the environment provides n-dimensional state with x q i being the input state of qubit q, and for the CartPole, Acrobot, and LunarLander environments, the dimension of the input state also corresponding to the number of qubits are n = 4, 6, and 8, respectively. The input state x q i is encoded by the angle of a single-qubit rotational gate U(x q i ). In the parameters layer, we use the trainable parameters θ q i to control single-qubit gates and the circuit has no entangling gates. In the output layer, the output of each qubit in the circuit is reused (copied) times and then all outputs are connected with classical NN. y j iq is the jth copy of the output of the qth qubit at step i, yiq. The duplicated outputs y j iq with q = 1, . . . , n and j = 1, . . . , are fed into the classical fully-connected NN to produce the scaled circuit outputs y ip with p = 1, . . . , k. Finally, we add the Softmax function to transfer the scaled circuit outputs to probability distribution. measurements, connection with a classical NN, and output reuse strategies. First, the expectation value of the measurements is obtained from Eq. (3) denoted by where q = 1, . . . , n is the indexes for different single qubits in the quantum circuit, Z q is the Pauli Z matrix of qubit q, and U(x q i , θ q i ) ∈ C 2×2 is a single-qubit unitary operator. Since the SVQC outputs merely come from the expectation values of eigenvalues on the unitary family {U † (x q i , θ q i ) Z q U(x q i , θ q i )}, the following technical strategies can enrich the expressive power of its outputs. Second, we link the single-qubit-based quantum circuit with a fully-connected NN layer to increase the expressive power of the quantum circuit so that it is more likely to achieve the optimal result in the practical optimization process [48,83]. Given that there exist optimal circuit outputs y * ip (x q i , θ q i ) ∈ R for variety tasks on OpenAI Gym, our target is to minimize |y ip − y * ip |, where y ip ∈ R is the scaled quantum circuit output for p = 1, . . . , k: where W pq ∈ R is the trainable parameters (weights) in the NN, b ip ∈ R are the biases, k is the number of actions and n is the number of qubits. Comparing the domain of y iq with that of y ip in Eqs. (5) and (6), the latter can increase the expressive power of the circuit output according to the studies in Ref. [48,83]. Finally, we duplicate the expectation value of the measurement times and then all outputs are fed into the classical fully-connected NN layer shown in Fig. 2. The final scaled output with the duplicated qubit outputs reads where kj is the expected weights with duplicated outputs, n ∈ N denotes the nth qubit, and ∈ N is the number of the output reuse. It will be shown later that the method of reusing qubit outputs improves the sample efficiency.

IV. NUMERICAL RESULTS AND DISCUSSION
To improve sample inefficiency in OpenAI Gym tasks, we compare the learning curves of SVQC, MVQC, and classical NNs on different tasks. Moreover, we use the IBM quantum devices to test the CartPole, Acrobot, and LunarLander tasks for comparing the real devices with an ideal simulator. In the following, the detailed discussion of simulator results are shown in Section IV A and  Fig. 2 are employed to complete the different RL tasks. The initial state is set to be in the ground state while the input layer is composed of a Hadamard gate followed by the gate sequence of (Ry(x q i )-Rz(x q i ), where q = 1, . . . , n are the qubit indexes. The parameterized single-qubit gate is Ry(θ q i ). The number of qubits for the CartPole, Acrobot, and LunarLander environments are n = 4, 6, and 8, respectively. Then the repeated measurements of σ q z are performed in the output layer. Finally, the measurement outputs could be reused for certain times and then connected with a fully-connected classical NN layer. The numbers appearing in the parentheses of FCN and CNN are the numbers of nodes in the fully-connected classical NN layers.

Environment
Learning algorithm Architecture (qubit number n) Times of reuses ( ) Number of actions (k)  Table IV.

A. Results on simulator
The training processes of SVQCs in the CartPole and Acrobot tasks are attained through Algorithm 1, while both Algorithm 1 and Algorithm 2 are used in the LunarLander-v2 task. The performances for different numbers of the reused outputs for the CartPole environment is shown in Fig. 3, indicating that the method of reusing qubit outputs improves the sample efficiency. The learning curves of the various models on the tasks of CartPole, Acrobot, and LunarLander are shown in Fig. 4 (d)-(f), respectively.
For the CartPole task, we compare the sample efficiency of single-qubit VQC (SVQC), multi-qubit VQC (MVQC), classical FCN, and CNN in Fig. 4 (d). Our work uses the SVQC architecture shown in Fig. 2 and described in Table III. The MVQC model for RL is discussed in some detail in tensorflow-quantum tutorial and the circuit architecture of MVQC used here is from [48]. Descriptions about the architectures of classical FCN and CNN used here can be found in Table III. From Fig. 4 (d), we find that SVQC (thick solid line) achieves maximum rewards in about 150 episodes. In comparison, MVQC (dashed line) converges around 500 episodes without achieving the maximum rewards while FCN (8,800) (dash-dot line), FCN (357) (dotted line), and CNN (173) reach the maximum rewards in about 400, 1,600, and 1650 episodes, respectively. Note that the numbers in the parentheses of different models are the total trainable parameters used in these models, respectively. These results provide concrete numerical evidence that SVQC could improve sample efficiency by about three times compared to classical FCN (8,800) under the same optimization process.   Fig. 4 (f). The figure shows that SVQC (solid line) achieves the average rewards of about 220 in 1,000 episodes. In comparison, FCN (dashed line) reaches the average rewards of 200, slightly lower than SVQC, in 1,750 episodes. These results conclude that the SVQC improves the sample efficiency by about two times compared to classical FCN (9,000) under the same optimization process. We would like to emphasize that this is the first time that the more complex control task of LunarLander can be achieved in the quantum RL field.
In conclusion, our proposed SVQC achieves higher re- wards than existing VQC-based models [47][48][49]84] and improves the sample efficiency (speed of convergence) compared to the classical fully-connected neural networks in the CartPole and Acrobot tasks. Moreover, our SVQC uses much (at least one order of magnitude) fewer trainable parameters with even better learning performance than the best performing FCNs shown in Fig. 4 (d)-(f).

B. Implementation on IBM quantum devices
In this section, we feed the trained parameters of our SVQCs into the IBM quantum devices, and compare their performance with that of the ideal simulators. The chosen benchmarking environments are CartPole-v0, Acrobot-v1, and LunarLander-v2 in OpenAI Gym. The real quantum device experiments for the Cart-Pole and Acrobot tasks are executed on "ibm lagos" and "ibm belem", while the LunarLander task is on "ibm lima", "ibm jakarta" and "ibm toronto".
The cumulative rewards on the CartPole-v0, Acrobot- v1, and LunarLander-v2 tasks are shown in Fig. 6 (a)-(c) and the numbers of measurements for these tasks are 1024, 1024, and 8192, respectively. Details of the used hyperparameters can be found in Table. V, and we will elaborate on the circuit architectures and their performances below.
We employ the quantum circuit in Fig. 5 (a) to conduct the task of CartPole-v0, where only a single qubit initialized in the ground state, |0 , is required. The input layer consists of a Hadamard gate and the sequence of (Rz, Ry, Rz) gates; while the parameter layer contains only an Rx gate. We import four classical trained parameters to scale the circuit output and use a trainable parameter for the angle of Rx. We upload the trained model parameters of the SVQC model in Fig. 5 (a) to IBM Quantum devices. According to Fig. 6(a), we find that the average reward of the real device over five tests is 190. This reward is comparable with the average reward, 200, obtained on the ideal simulator.
The six-qubit quantum circuit shown in Fig. 5 (b) is used to complete the Acrobot-v1 task. Similarly, the initial state is set to be in the ground state while the input and parameter layers are composed of a Hadamard gate followed by the gate sequence of (Ry, Rz, Ry). Then the repeated measurements of σ z are performed in the output layer. This circuit consists of six trainable rotational angles and 21 parameters for output rescaling. The average reward obtained from the quantum machine is -90 while the value from the ideal simulator is -84. Again, the rewards of the real device and the idea simulator are comparable.
The first quantum RL agent based on SVQC consisting of a 24-qubit quantum circuit with the architecture demonstrated in Fig. 5 (c) is used to complete the LunarLander-v2 task on the real quantum device. The circuit inherits the architecture of SVQC used in the Acrobot-v1 task and is further expanded to a larger scale by introducing 24 trainable angle parameters and 100 output scaling parameters. By comparing the average reward of the real device with that of the ideal simulator in Fig. 6 (c), we find the average reward of the real device is 200, slightly worse than the average reward of 250 of the ideal simulator. We expect that conducting more test runs will further increase the average reward of the real device, but we did not continue the experiments on IBM devices due to the cost of the expensive quantum computational resources.
To the best of our knowledge, it is the first time the complex RL tasks can be accomplished on IBM quantum devices. The results demonstrate that even though the training is on the quantum simulator without noise, the trained SVQC models have similar performances on current NISQ devices compared to the ideal simulator.

V. CONCLUSION AND FUTURE WORK
The SVQC performs better than previous studies [47][48][49]84] which use several CNOT or CZ gates on the Cart-Pole and Acrobot environments. This leads to an open question of "What are the roles of entangling gates in MVQC for the RL tasks? " On the other hand, we find SVQC with the output reuse can solve the RL tasks more efficiently than the classical NNs. This brings the question of "What is the quantum-inspired algorithm, which can solve the RL problems efficiently? " Since SVQC can be implemented on the current NISQ quantum devices to handle classical control and box2d tasks in openAI Gym. Therefore, "What is the limitation on the current NISQ devices in RL tasks? " is the remaining question for the future work.

State preprocess layer:
There are three main encoding strategies of the quantum circuit: basis, amplitude [27,28], and Hamiltonian encoding schemes. The basis encoding scheme needs the runtime of O(M N ) for state preparation without QRAM [86], while the amplitude and Hamiltonian encoding schemes can reduce the time to O(log(M N )) with QRAM. Moreover, the Hamiltonian encoding tries to build the kernel space, which is hard to be built using classical computers [87].

Parameter layer:
There are different technologies to design the different architectures of the circuit by machine learning [88] or reinforcement learning [89,90].

Measurement:
The general quantum circuit output is σ z = tr (ρ(x i , θ i ) σ z ), where ρ(x i , θ i ) ∈ C 2 n ×2 n is the density matrix depending on parameters and input data, and σ z ∈ C 2 n ×2 n is the projective matrix. A challenge about the measurement is that lots of shots would eliminate the runtime advantage [91,92]. There are strategies to improve the efficiency in measuring the quantum state [62,93].

Optimization:
The challenge about circuit optimization lies in barren plateau [94]. The gradient of parameters would vanish exponentially in the optimization process. Using tree structure [95], tuning the parameters with an iterative optimization structure, and using adaptively selected Hamiltonian [96] can mitigate the barren plateau in the process.

Discussion of VQC-based quantum reinforcement learning
There are many technical skills in VQC-based quantum reinforcement learning. References [48][49][50] provide various methods to solve the OpenAI Gym tasks. The methods can be divided by the circuit architecture that consists of the input, parametric, and output layers.
In the input layer, the additional trainable parameters are encoded by the rotational angles of the gates that improve the performance on the Cartpole and Acrobot tasks [48,49]. For the input and parametric layers, the repeated application of re-uploading enhances the performance on the classical control tasks [48][49][50]. In the output layer, introducing the extra trainable parameters to rescale the measurement outcomes [48,49] or adding the classical neuron network connection improves the cumulative rewards [50] on the Cartpole and Pendulum tasks. The followings are the constraints on the Cartpole, Acrobot, and LunarLander environments. The number of states of Cartpole , Acrobot, and LunarLander are four, six, and eight, respectively, and the numbers of actions are two, three, and four, respectively. The constrains on different observations (states) of Cartpole and Acrobot are subsequently shown in Table. VI and Table. VII.
The detailed information of LunarLander is as follows. According to the description of the environment in Ope-nAI Gym, the reward for moving from the top of the screen to the landing pad and zero speed falls between 100 and 140 points. The episode finishes if the lander crashes or comes to rest, receiving an additional reward of −100 or +100 points. Each leg ground contact is +10. The reward is −0.03 for firing the side engine, and −0.3 for firing the main engine each frame.