{'fraction_non_alphanumeric': 0.023087825172587208, 'fraction_numerical': 0.0055776528139715635, 'mean_word_length': 5.008377970059057, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RL-EA: A Reinforcement Learning-Based Evolutionary Algorithm Framework for Electromagnetic Detection Satellite Scheduling Problem

The study of electromagnetic detection satellite scheduling problem (EDSSP) has attracted attention due to the detection requirements for a large number of targets. This paper proposes a mixed-integer programming model for the EDSSP problem and an evolutionary algorithm framework based on reinforcement learning (RL-EA). Numerous factors that affect electromagnetic detection are considered in the model, such as detection mode, bandwidth, and other factors. The evolutionary algorithm framework based on reinforcement learning uses the Q-learning framework, and each individual in the population is regarded as an agent. Based on the proposed framework, a Q-learning-based genetic algorithm(QGA) is designed. Q-learning is used to guide the population search process by choosing variation operators. In the algorithm, we design a reward function to update the Q value. According to the problem characteristics, a new combination ofis proposed. The QGA also uses an elite individual retention strategy to improve search performance. After that, a task time window selection algorithm is proposed To evaluate the performance of population evolution. Various scales experiments are used to examine the planning effect of the proposed algorithm. Through the experimental verification of multiple instances, it can be seen that the QGA can solve the EDSSP problem effectively. Compared with the state-of-the-art algorithms, the QGA algorithm performs better in several aspects.


Introduction
In recent years, the rapid development of aerospace has provided new solutions for various types of tasks such as information communication, environmental monitoring, and disaster forecasting [1]. The applications that satellites can play can be divided into three categories: observation, communication, and navigation according to the different carrying payloads. Observation satellites can be further classified according to the payload equipped with visible light, infrared, synthetic aperture radar, antenna, and others. Satellites that use signal receivers and antennas as payloads are called electromagnetic detection satellites. Electromagnetic detection satellites detect and process electromagnetic signals to obtain helpful information. Compared with optical imaging satellites, electromagnetic detection satellites have a wider detection range and are not easily affected by weather factors. It can also generate more stable and reliable detection profits. Based on detecting targets with known signal characteristics, electromagnetic detection satellites can also search for unknown signals through a wide range of frequency bands.
There are a large number of electromagnetic signals on the surface of the earth where we live, and many electromagnetic signals together constitute the surface electromagnetic environment. The detection of surface electromagnetic environment can effectively support the work in various fields such as environment, agriculture, military, meteorology, and others. The electromagnetic detection satellite scheduling problem (EDSSP) is to plan the satellite resources used to execute the demand and determine the specific execution time of each task. Detection requirements are put forward by various types of users and hope to be effectively satisfied. However, because the satellite is flying in a fixed orbit, only a part of the tasks are within the detection range of the satellite, and limited by the capabilities of its payload, the tasks that can be performed during each lap are limited. The goal of electromagnetic detection satellite scheduling is to generate a reasonable task sequence within a certain time range. This sequence can generate sufficient detection profit as much as possible and satisfy users' preferences. The phenomenon of oversubscription is very common in satellite task scheduling problems, whether in electromagnetic detection satellite task scheduling problems or in optical imaging satellite task scheduling problems, satellite data download scheduling problems and other types of satellite scheduling problems.
Electromagnetic signal detection needs to effectively balance the relationship between the detection accuracy and the satellite's capability level. From the signal detection activity itself, when the detection points are increased to improve the detection accuracy, the amount of data generated by the detection will also increase accordingly. The capacity of the satellite's onboard storage device is limited. If the detection task data has been fully used by the satellite, the detection task cannot be continued until the data is erased. The reasonable setting of detection accuracy according to user needs can improve detection performance as much as possible based on utilizing satellite resources.
The existing research on the scheduling of electromagnetic detection satellites has designed corresponding solutions for multiple areas detection, moving target detection, and other scenarios [2,3]. However, the simplified way of dealing with satellite working modes only solves the common problems of all kinds of satellite scheduling problems. It can be said that the research is not mature and in-depth enough. Compared with the electromagnetic detection satellite scheduling problem, the researches on the optical satellite observation scheduling problem are more [4,5]. In terms of model building, Berger and Barkaoui used the quadratic programming method to construct the visible light satellite task scheduling model [6,7]. Cho et al. proposed a two-step binary linear programming formulation to formulate a satellite observation scheme for a low-orbit satellite constellation [8]. Niu et al. considered using satellites to complete post-disaster large-area observation tasks built a multi-objective optimization model, and used genetic algorithms to solve this problem [9]. Chen et al. proposed a mixed-integer programming model for agile satellite scheduling, constructing decision variables based on conflicts between time windows and using 5-index variables to obviate the need for a Big-M approach [10].
Evolutionary algorithms have been widely used in solving satellite task scheduling problems. The applications of the genetic algorithm [11], differential evolution [12], ant colony algorithm [13], meme algorithm [14] and others algorithms have increased rapidly in recent years. Compared with accurate algorithms, the evolutionary algorithm has a lower computational cost on large-scale problems. The studies of [15,16] show that genetic algorithms are more suitable for solving satellite task scheduling problems than other evolutionary algorithms. E. et al. proposed an individual reconfiguration-based integer coding genetic algorithm (IRICGA) for regional target observation planning and designed a new area partitioning method based on two kinds of discrete parameters [17]. Zhang and Xing adopted a novel idea of encoding and decoding for the integrated satellite imaging and data transmission scheduling problem to improve the search efficiency of the genetic algorithm. Two test cases are used to verify the algorithm effect [18]. Song et al. and Du et al. used nondominated sorting-based and decomposition-based multi-objective evolutionary algorithms, respectively, to find high-quality solutions for satellite range scheduling problems [19,20]. Xhafa et al. proposed a genetic algorithm combined with the STK solution tool to obtain a plan of ground stations [21].
Existing research on reinforcement learning to solve satellite scheduling problems rarely considers combining with combinatorial optimization methods. Research using reinforcement learning methods alone to solve this problem is also in its infancy. For example, Wei et al. directly used reinforcement learning methods to solve imaging satellite scheduling problems [22]. Chen et al. built an end-to-end reinforcement learning framework, used the attention mechanism, and proposed an actor-critic network training method [23]. To the best of our knowledge, existing research has not employed reinforcement learning embedded in an evolutionary algorithm framework to solve the satellite task scheduling problem.
Combining evolutionary algorithms with reinforcement learning can effectively utilize the respective advantages of the two methods to improve search efficiency and search performance [24]. Existing research mainly focuses on the numerical optimization problem [25,26]. These combined methods achieve good performance. For the EDSSP problem, we try to use a new combination of an evolutionary algorithm and reinforcement learning method to obtain high-quality solutions. Control parameters are extremely important for evolutionary algorithms, and these parameters are always sensitive to the problem. This easily affects the generalization ability of the algorithm. Reinforcement learning methods have strong generalization capabilities but are prone to poor performance due to a lack of domain-specific knowledge. So, we try to apply the reinforcement learning method to the evolutionary algorithm framework and effectively combine the respective advantages of these two methods. Our proposed evolutionary algorithm framework based on reinforcement learning (RL-EA) can allow researchers to invest more energy in algorithm and strategy design. The RL-EA framework can be combined with various evolutionary algorithms. In this paper, we design a Q-learning-based genetic algorithm based on the framework. In QGA, we use reinforcement learning methods to help evolutionary algorithms improve generalization and obtain highquality solutions. This is the first time that reinforcement learning is used as a component embedded in an evolutionary algorithm for a satellite scheduling problem.
The innovations of our research are: 1. A refined model of the EDSSP problem of mixed-integer programming is built. The model considers many practical factors such as detection mode, detection angle, and data volume, which is conducive to obtaining a more practical scheduling scheme.
2. An evolutionary algorithm framework based on reinforcement learning and a genetic algorithm based on reinforcement learning are proposed. In the framework, the reinforcement learning method is used to drive the change of the evolutionary algorithm. After that, a Q-learning-based genetic algorithm on the basis of the framework is proposed. Q-learning is used to choose variation operators of QGA. In QGA, the reward function, the Q value update strategy, and the strategy selection method are designed. In order to calculate fitness function values, a task time window selection algorithm is used to generate detection plans. It can also decide the execution time of each task.
3. The effectiveness of the proposed QGA algorithm is verified by the EDSSP problem. It provides a new algorithm design and solution method to the scheduling problem.
The remainder of this paper is organized as follows. The second part of this paper introduces the electromagnetic detection satellite scheduling problem model. The third part of this paper introduces the evolutionary algorithm framework based on reinforcement learning and the Q-learning-based genetic algorithm. The fourth part of this paper verifies the effect of the proposed algorithm. The fifth part of this paper summarizes the content of the paper and gives future research directions.

Problem Description
The EDSSP problem is to designate a time-ordered task execution sequence for electromagnetic detection satellites, which maximizes the detection sequence profits as much as possible based on satisfying various satellite constraints. To ensure the execution of the task, it is not only necessary to determine the on-off time of the receiver, that is, the start time and end time of the task, but also to set a series of parameters such as detection mode, frequency, bandwidth, and polarization mode.
The time range from the beginning to the end of the signal beam coverage of the electromagnetic satellite is called the visible time window. Since the electromagnetic satellite antenna can effectively detect a wide range of ground signals, to reduce noise and improve the detection accuracy, the angle between the signal source and the pointing direction of the satellite antenna needs to be within a certain range.
The detection profit of electromagnetic detection satellites is affected by two factors. On the one hand, the signal detection profit is affected by the angular relationship between the satellite antenna and the signal source. When the center of the satellite antenna passes directly above the signal source, the maximum signal gain can be obtained, and the signal gain is directly linked to the detection effect. Signal gain is inversely related to the angle between the antenna and the signal source. In other words, when the centerline of the signal source beam coincides with the extension line of the satellite antenna pointing direction, the detection effect is the best. On the other hand, the detection accuracy will also affect the effect of signal detection. The detection accuracy is limited by the inherent capability of the receiver, and the number of sampling points will also have a direct impact on the detection accuracy. There is a one-to-one correspondence between the bandwidth and the number of sampling points, and the amount of data obtained by different detections with the number of sampling points is also significantly different. The limited storage capacity of satellites means that only some tasks can be set to the highest bandwidth, while many tasks need to be set to a smaller bandwidth.

Symbols and Variables
T : Set of tasks, a total of |T | tasks. For task j, the following attributes are defined: est j : The earliest allowed start time of the task let j : The latest allowable end time of the task dur j : Task request duration θ max j : Maximum allowable detection angle of the task degree j : Task importance level m j : The amount of data for the task G j : The signal gain can be obtained from the task S : Set of satellites, a total of |S| satellites. For satellite s i , the following attributes are x ijok : Whether the satellite i is in the time window k on the orbit o whether the task j is executed, if the task is executed, st ijo : Start time of the satellite i on the orbit o to execute the task j;

Mathematical Model
Assumptions: • All electromagnetic sounding satellites have the same structure and function; • The detection process will not be affected by external factors; • The detection task is definite, and there will be no temporary changes or cancellations; • The satellite has sufficient energy during orbit; • Each task can be completed after one detection, without repeated detection.
The calculation formula of the detection profit that can be obtained by a single detection task is: where u = 2.07123 sin (θ) / sin (θ 3dB ), J 1 (u) and J 3 (u) are the 1st and 3rd order Bessel functions of the first kind, respectively. θ is the angle between the satellite antenna and the center of the signal source, θ 3dB is the angle at which the antenna gain is attenuated by 3dB relative to the center of the beam, and the calculation formula as follows.
where λ represents the wavelength, D indicates the diameter of the antenna. In this paper, the bandwidth used by satellites to perform detection tasks is dynamically matched according to the priority of detection tasks. The importance of the task is high, and the bandwidth used is large so that the detection effect will be better. However, due to the limitation of satellite storage, the detection bandwidth needs to be scientifically set. The formula for setting the bandwidth according to the degree of importance degree j is shown below.
The bandwidth setting also affects the detection profit, and the resulting gain can be measured by the function Ω [ϕ (degree j )]. When the bandwidth of the detection task j is set according to ϕ (degree j ), the amount of data generated per second is β · ϕ (degree j ). Combined with the task detection time dur j , the total data amount of task j can be obtained by: The electromagnetic detection satellite needs to adjust the parameters of the onboard equipment when performing different tasks. The transition time between tasks is composed of four parts. The first part is the time required for the change of the polarization mode, and the second part is the time required for the change of the detection mode. The time used, the third part is the frequency band transition time, and the fourth part is the bandwidth transition time. In addition, it also takes a certain amount of time for each onboard equipment to be turned on and off, and this time interval must be satisfied between the two tasks. To simplify the constraint judgment, we introduce the variable tran ijj , which represents the total transition time. The transition time of the two tasks j and j is as follows.
where Γ pol is the satellite polarization transition time, Γ mode is the satellite detection mode transition time, Γ band is the satellite bandwidth setting transition time, Γ f re is the satellite frequency band transition time, ∆ is the satellite load on/off time The scheduling goal of the EDSSP problem is to obtain the highest possible detection profit. The objective function is represented as follows.
Objective function: where G j is signal gain can be obtained from the task, Ω [ϕ (degree j )] is the gain due to the bandwidth setting. Constraints: Constraints 1-2 indicate that the task start and end must be within the time frame required by the task. Constraint 3 indicates that the detection angle cannot exceed the task maximum angle requirement. Constraints 4-5 indicate that the task start and end need to be within the visible time window. Constraint 6 indicates that the satellite cannot exceed the upper limit of the satellite storage capacity in each orbit. Constraint 7 indicates that the satellite must meet various transition time requirements to perform both tasks. Constraint 8 indicates the limit on the number of probes for the task. Constraint 9 indicates the value range of the decision variable.

The Proposed Method
Since the single satellite imaging scheduling problem is NP-hard, more complex EDSSP problems do not have a polynomial-time algorithm [27,28]. Therefore, we propose an evolutionary algorithm framework based on reinforcement learning to solve the EDSSP problem. The framework proposes a new reinforcement learning method that can be used to improve the search performance of the evolutionary algorithm. After that, we propose a Qlearning-based genetic algorithm according to the algorithm framework. In addition, we also design a task time window selection algorithm for generating schemes. The evolutionary algorithm framework, Q-learning-based genetic algorithm, details in evolutionary algorithms, and strategies in reinforcement learning and task time window selection algorithms are given in this section.

Reinforcement Learning Based Evolutionary Algorithm Framework(RL-EA)
Evolutionary algorithms simulate the evolution process of biological populations and find high-quality solutions to problems in the form of population evolution. Population search brings a strong global search capability to evolutionary algorithms, which ensures its performance in solving large-scale complex problems.
The obvious disadvantage of evolutionary algorithms is that the search results are extremely sensitive to the parameter configuration in the algorithm. Parameter-sensitive features make many evolutionary algorithms have certain dependencies on problems and scenarios. When problems or scenarios change, parameters need to be adjusted or reset. The parameter setting process is likely to be time-consuming. In order to enhance the versatility of evolutionary algorithms in different scenarios, the workload of setting algorithm parameters is reduced. We use the Q-learning method to drive the algorithm to update autonomously and let the evolutionary algorithm select an appropriate search strategy based on the search performance to drive the population evolution. Without loss of generality, we present a general evolutionary algorithm framework based on reinforcement learning. The algorithm framework can be flexibly adjusted according to different evolutionary algorithms in actual use. The pseudo-code of the evolutionary algorithm framework based on reinforcement learning is shown in Algorithm Table 1.
The evolutionary algorithm framework based on reinforcement learning selects the appropriate action according to the state in the iterative search process of the population, chooses one action to obtain a new generation of the population, evaluates the performance of the population and the income after adopting the action, and updates the state for a new round of population evolution. Population evolution and state update are continuous in the process of population search, and there exists frequent information exchange between them.
As a general framework, population evolution can be flexibly set according to the specific evolutionary algorithm used and the problems to be solved. Input parameters, operators, control parameters, and other strategies in the algorithm may also be different according to the specific algorithm used. For example, the genetic algorithm uses control parameters to limit crossover and mutation, while the ant colony algorithm uses control parameters to limit pheromone update.

Q-learning based Genetic Algorithm (QGA)
According to the evolutionary algorithm framework based on reinforcement learning, we design a Q-learning based genetic algorithm based for the EDSSP problem. Genetic algorithm has good global search ability, and it has a good performance in the field of sequence optimization and scheduling, such as traveling salesman problems, vehicle route planning problems, satellite task scheduling problems, and many other problems. To improve the local search ability of the genetic algorithm, we introduce the elite individual retention strategy into the genetic algorithm. QGA uses the Q-learning method to complete the variation of the population. The variation includes two genetic operators, crossover and mutation. Crossover operation has a variety of crossover methods. The population obtained by the QGA algorithm uses a task time window selection algorithm(TTWSA) to complete the decoding and generate the task execution plan. The profit of the task execution plan obtained by the task time window selection algorithm is used for the reward calculation and then the Q value update process. In other words, TTWSA provides a basis for the learning method to select new actions. The pseudo-code of the genetic algorithm based on Q-learning is shown in Algorithm Table 2.
In QGA, count is a parameter to control whether the algorithm executes elite individual retention. When the number of records is equal to T hre (Line 22), elite individuals are no longer retained. Obtaining the task planning scheme from the individual genetic algorithm needs to use the task time window selection algorithm to determine the tasks that can be executed and specify the specific start execution time and end time.

Individual Representation and Initialization
Initialization is the first step in the evolutionary algorithm population search process. It needs to construct an initial solution in a certain way and further search to find a higherquality solution on the basis of this solution. Individual representations can be encoded in real numbers, which correspond one-to-one with the sequence numbers in the task sequence. Decoding uses the task time window selection algorithm to determine whether to execute the task according to the serial number. If it is executed, specify the specific execution time, or abandon the execution of the task. The final task execution plan is obtained by arranging it in sequence. The advantage of real number coding is to ensure the unique correspondence between the elements at each position in the sequence and the task, and the decoding process is simple. Another benefit of this individual representation method is that it effectively guarantees the legitimacy of the solution structure without the need to use any repair methods.

Fitness Evaluation
Fitness evaluation has two functions in the QGA algorithm, one is to select individuals from the population, and the other is to evaluate the reward update results in the Q-Learning method. The fitness evaluation is obtained by calculating the objective function value of the individual corresponding to the plan by Eq. 7. From the individual to get the plan is obtained by the task time window selection algorithm described in section 3.3.

Individual Selection
Individual selection is the premise of population evolution. After an individual is selected, the corresponding operation can be completed to obtain a new individual. On the one hand, individual selection should make good performers more likely to be successfully selected, and on the other hand, other individuals should also have a certain possibility of being selected so as to ensure the diversity of algorithm optimization. Therefore, we use a roulette approach to determine individual choices for evolutionary algorithms. The calculation formula for selecting individuals in roulette is as follows.
where f it i represents the fitness function value of individual. The fitness function value is calculated using the Eq. 7.

Crossover
The purpose of evolution is to improve the performance of individuals within a population. We use crossover and mutation to achieve individual evolution within the population. The two are collectively referred to as variations. For the crossover operation, we set up crossover operators with various lengths of partial sequence lengths, which are called short segment crossover operator (denoted as C1), medium segment crossover operator (denoted as C2), and long segment crossover operator (denoted as C3), segment flip (denoted as C4), Foremost position crossover (denoted as C5), segment internal order adjustment based on earliest allowed start time (denoted as C6), based on task requirements Fragment internal order adjustment for execution time (denoted as C7).
Short segment crossover operator (C1): Short segment crossover refers to selecting two task sequence segments of length L from an individual and exchanging the positions of the two segments in the individual. The schematic diagram of the short segment crossover is shown in Figure 1. Long segment crossover operator (C3): Long segment crossover refers to selecting two task sequence segments of length 3L from an individual and exchanging the positions of the two segments in the individual.
Fragment flipping (C4): Fragment flipping is to select a task sequence fragment of length L from an individual and rearrange the task sequence from back to front. The schematic diagram of the fragment flipping is shown in Figure 2. Segment internal order adjustment based on earliest allowed start time (C6): Select a task segment of length L from an individual and adjust the internal segment order from front to back according to its task's earliest allowed start time est j .
Segment internal sequence adjustment based on task duration (C7): Select a task segment of length L from an individual, and adjust the internal segment sequence from short to long according to the task required duration dur j .

Mutation
Compared with crossover, the variation of mutation is smaller, which is also helpful for improving the search performance of the algorithm. The mutation is performed by exchanging two positions in the sequence. We combine the crossover operator and mutation operator to complete the individual evolution of the population. After the change is completed, the fitness is evaluated, and the fitness value is calculated by using the Eq. 7 and the task time window selection algorithm to calculate the plan after the individual has been arranged. In the QGA, the crossover and mutation methods used for optimization are evaluated by the Q-learning method, and the operation used for individual evolution is selected according to the Q value.

Reinforcement Learning
We adopt the Q-learning method in reinforcement learning to guide the search for evolutionary algorithms. Q-learning is a simple and effective automatic evaluation mechanism, which is widely used in various models such as DQN and DRL according to the actions taken by the algorithm adaptive decision-making according to the Q value [29,30]. Q-learning consists of five parts: action, state, reward, learning algorithm, and environment, which can be represented as a quadruple of A, S, R, L, E [31]. The Q-learning method selects the population update strategy through the update of the Q value, which replaces the traditional operator selection method in the evolutionary algorithm. We regard each individual in the population as an agent and select and evaluate the search strategy through the Q value of the agent's performance. The interaction between Q-learning and the environment is evaluated by the fitness function. The calculation of the fitness function value requires the selection of an algorithm according to the task time window. The algorithm will be introduced in the next section. The running process of Q-learning is shown in Figure 3. The formula for calculating reward is shown below.
where F t and F t−1 are values of fitness at the time t and t − 1, respectively. The updated formula for the Q value is shown below.
where α is the learning rate, and γ is the discount factor. An important part of Q-learning is the Q-value table, which records the performance values of all crossover and mutation combinations used in the search process. When the evolutionary algorithm framework contains n c kinds of crossover operators and n m kinds of mutation operators, the total number of actions contained in the Q-value table is n c × n m + n c + n m . In our proposed framework, there are 7 kinds of crossover operators and 1 kind of mutation operator. Therefore, the number of actions that each agent can use is 15. We set the state to two levels according to the fitness function value change, which is higher (denoted as I) and lower (denoted as II). In the proposed algorithm, the number of <state, action> combinations is 30. For each agent, the probability of selecting a j action in state s i is usually determined by Softmax strategy: where Q t (s i , a j ) is the corresponding value in the Q table at time t, and T is a control parameter.
Due to the large value of the objective function in the EDSSP problem, the introduction of the control parameter T can effectively prevent the probability value of individual individuals from being too large, which makes it difficult for some individuals to be selected. At the same time, Q-learning needs to balance the exploration and exploitation of algorithms [10]. In order to prevent the Q-learning algorithm from converging prematurely and falling into a local search and being unable to jump out, we use ε. When the randomly generated probability value is less than ε, a random change strategy is selected as the individual evolution of the individual evolution in this population.

Elite Individual Retention Strategies
We use an elite individual retention strategy in the QGA to speed up the algorithm's convergence. When the retention process occurs, the optimal individual obtained by the search is retained in the next generation population. The use of elite individuals will affect the global search ability of the algorithm after a certain period of search. Therefore, we introduce a parameter count and a threshold T hre. When the current population search is not improved compared to the previous generation population, that is, the maximum fitness value of the population does not change, the parameter count will increase by 1. If the value of parameter count is equal to the threshold T hre, it is no longer reserved for elite individuals.

Algorithm Termination
After a certain number of iterative searches, the algorithm needs to end the operation and output the optimal result as the final task execution plan. Since we adopt the crossover and mutation method based on reinforcement learning, when Q-learning selects an action, it can correspond to one or two evolution operations. Therefore, it is reasonable to use the number of evolutions num eval as the evaluation index to record the iterative search of the genetic algorithm and the termination of the algorithm. When the algorithm evolution times reach the set maximum evolution times M F E, the algorithm search will be stopped.

Task Time Window Selection Algorithm
When the task execution is determined through algorithm optimization, the start time and end time of the task need to be specified. The setting of task start time and end time not only affects the detection effect of the task but also affects the satisfaction of various constraints of the problem. We propose a two-stage task time window selection algorithm with initial screening and then specifying a time. The purpose of preliminary screening is to ensure that the task execution can meet the requirements of various constraints of the detection task, such as angle, transition time, and other constraints. Therefore, the preliminary screening needs to clip some time windows that do not meet the constraints or discard the current time window and try to schedule tasks in the next time window. The basis of preliminary screening is to obtain the actual earliest actual allowable start time AEV T ijok and the latest actual allowable completion time ALV T ijok after cutting according to the Eq. 10-12. The formulas of AEV T ijok and ALV T ijok are shown as Eq. 21 and Eq. 22 respectively.
After the time window is trimmed, the start time and end time of the detection task need to be specified. Before determining the task execution time, it is necessary to check the new time window after cutting and arrange the task after passing the check.
Generally speaking, the signal gain that the satellite can obtain from the signal source follows the law of first increasing and then decreasing gradually. In order to obtain more signal gain, the detection process should ensure that the angle between the antenna and the signal source is as small as possible. The time frame in which the task is to be performed should be chosen so that the detection can be as profitable as possible. Generally speaking, the change of the angle between the satellite and the target in the time window can be regarded as a symmetrical process. Therefore, the middle position of the cropped time window is selected as the middle moment of the detection task, which is usually the moment when the detection angle is the smallest, and the start execution time and the task end time are determined according to this position. The formula for calculating the best start execution time is as follows.
where bst ijo represents the best start time of the execution task j of satellite i on orbit o.
If this moment can meet the requirements of other constraints, it will be regarded as the final start time; otherwise, move the start time backward until all constraints are met. After arranging all the tasks in order, the execution plan of the detection task is obtained. The pseudo-code of the task time window selection algorithm is shown in the Algorithm Table  3.

Experimental Settings
The configuration of the experiment in this paper is Core I7-7700 3.6 GHz CPU, 16 GB memory, Windows 11 operating system desktop computer, and Matlab 2020a is used for coding. All algorithms are run under the same system configuration.
We use a series of satellites for our experiments. The satellite orbit parameters can be defined by a hexagram: the length of semi-major axis (a), eccentricity (e), inclination (i), argument of perigee (ω), right ascension of the ascending node (ϕ), and mean anomaly (m). Table 1 gives the initial orbital parameters of two satellites. To verify the effect of our proposed algorithm. We choose an improved genetic algorithm (IGA) [32], an adaptive large neighborhood search algorithm (ALNS/TPF) [33], an artificial bee colony algorithm (ABC) [34], and a construction heuristic algorithm (CHA) as state-ofthe-art algorithms. The IGA algorithm adopts a heuristic mechanism and uses knowledge to guide the algorithm search. The ALNS/TPF algorithm combines tabu search and adaptive large neighborhood search algorithm to find high-quality solutions through destruction and repair. The ABC algorithm search solution space (also known as nectar sources) through three types of bees, imitating the way bee populations search. The CHA algorithm gives the planned task sequence according to the way of sorting the task profit. The above four algorithms have a large number of applications in satellite task scheduling problems and other planning problems and have good performance.
The evaluation times M F E of all algorithms were set to 5000 times. The control parameter T of QGA is set to 1000, the initial learning rate α is set to 0.01, the discount factor γ is set to 0.95, the threshold T hre is set to 100, the length L is 2 and ε is 0.01. The crossover probability of the IGA algorithm was set to 0.9, and the mutation probability was set to 0.1. The initialized score of ALNS/TPF is 100, and the increased scores according to performance are 50, 20, 10, respectively, and the tabu list length is 10. The population size N p of QGA, IGA, and ABC are all set to 10.
We design four scale instances, named small scale (100-300 tasks), medium-scale (400-600 tasks), large scale (700-800 tasks), and ultra-large-scale (1000-1400 tasks). Each instance is marked in the form of "A-B", "A" represents the number of tasks, and "B" represents the internal number of the instance under that number. Detection tasks are randomly distributed around the world. Each algorithm was run 30 times on an instance. The detection profit of the task sequence is the basis for the algorithm evaluation. For the evaluation benchmark metrics, we separately count the maximum (denoted as Best), mean (denoted as Mean), and standard deviation (denoted as Std. Dev.) of the obtained results and perform Wilcoxon rank-sum test on the results. In addition, the CPU time and convergence speed of the algorithm are also used to evaluate algorithms.  increases with the increase of the problem size. There is basically no difference in the results between all five algorithms except for the maximum profit for 100 tasks. When the task size increases to 200, the gap between the algorithms becomes significant. It is worth noting that when the task size is increased to 300, the maximum benefit that can be obtained from the search does not increase in multiples of the task size. This means that the influence of constraints on task scheduling increases, and some tasks that violate constraints cannot be executed.  mentioning that the HA algorithm with a simple structure can also search for a high-quality detection task execution plan. The experiments of ultra-large-scale instances can more effectively reflect the solving ability of the algorithm in the case of complex constraints and large solution space. As shown in Table 4, the ultra-large-scale scheduling problem effectively utilizes the advantages of reinforcement learning in the QGA. It can change the strategy selection to allow the algorithm to try new search spaces continuously. Meanwhile, the elite individual retention strategy improves the local search ability of the algorithm. The performance of the other four algorithms is relatively close on the whole, and ABC performs better than other stateof-the-art algorithms in most cases.  Figures 5(a), 5(b), 5(c) and 5(d) show the convergence speed of different algorithms at 300, 600, 800, and 1400 task scales, respectively. In a small-scale instance, when the number of fitness evaluations reaches 3000, the algorithm basically converges. When the task size is larger, more search algebras are required for the algorithm to converge. Compared with the compared state-of-the-art algorithms, the QGA algorithm has a faster algorithm convergence speed and has the ability to exploit continuously.

CPU Time Analysis
We make statistics on algorithms' CPU time. The results in Table 5 are meantime results. It can be seen from the results that the increase in time mainly comes from the increase in problem scale, and the time used for task time window selection increases significantly. When the task scale is small, the running time of the proposed algorithm is not dominant due to the existence of a series of operations of Q value calculation and action selection. When the task scale increases, the task sequence generated by the QGA algorithm in the front position has a high possibility of being successfully scheduled. When the time window resource cannot be scheduled, the task time window selection algorithm will not be run to try to schedule. So the computational resources consumed by the reinforcement learning in the algorithm can be ignored. It is not difficult to find that the ABC algorithm takes a lot of time, which is closely related to the complex division of labor within the population and the interaction of bees' search information. Since the environment of practical problems is often very complex, and the task scale is large, the proposed algorithm is competent for task scheduling. Instance QGA IGA ALNS/TPF ABC time. From the perspective of problem scale, the QGA algorithm has a good performance on large-scale problems, but its time advantage is not obvious in small-scale problems due to its Q-value evaluation and selection mechanism. How to make reinforcement learning perform the same or better effect with fewer evaluations is worth further research. Overall, the RL-EA algorithm proposed in this paper can solve the EDSSP problem very well and has a good application prospect.

Conclusion
In this paper, we propose a reinforcement learning-based evolutionary algorithm framework to solve the EDSSP problem. Reinforcement learning methods make algorithms smarter, taking effective actions based on their interactions with the environment during the search process to improve search performance. Based on the framework, we propose a Q-learning based genetic algorithm. The changes in the evolutionary algorithm are guided by reinforcement learning, which allows each individual to select an appropriate change strategy according to the reward and Q value. We construct a <state, action> combination method according to the search performance and update the corresponding value after each search is completed. The elite individual retention strategy used in the algorithm can improve the search performance of the algorithm. It can be seen from the experiments that our proposed algorithm has obvious advantages in solving the EDSSP problem.
In future research, we will try to reduce the computational cost of the Q-value evaluation mechanism to further accelerate the search speed without affecting the performance of the algorithm. More learning methods will be considered, such as transfer learning, deep learning, and others. In addition, we will also consider applying learning-based approaches to online scheduling problems. The environment faced by the online operation of the system is more complex, and the requirements for algorithm operation efficiency are higher.