{'fraction_non_alphanumeric': 0.029287131899372418, 'fraction_numerical': 0.008662768867671512, 'mean_word_length': 4.92986641221374, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Graph–sequence attention and transformer for predicting drug–target affinity

Drug–target binding affinity (DTA) prediction has drawn increasing interest due to its substantial position in the drug discovery process. The development of new drugs is costly, time-consuming, and often accompanied by safety issues. Drug repurposing can avoid the expensive and lengthy process of drug development by finding new uses for already approved drugs. Therefore, it is of great significance to develop effective computational methods to predict DTAs. The attention mechanisms allow the computational method to focus on the most relevant parts of the input and have been proven to be useful for various tasks. In this study, we proposed a novel model based on self-attention, called GSATDTA, to predict the binding affinity between drugs and targets. For the representation of drugs, we use Bi-directional Gated Recurrent Units (BiGRU) to extract the SMILES representation from SMILES sequences, and graph neural networks to extract the graph representation of the molecular graphs. Then we utilize an attention mechanism to fuse the two representations of the drug. For the target/protein, we utilized an efficient transformer to learn the representation of the protein, which can capture the long-distance relationships in the sequence of amino acids. We conduct extensive experiments to compare our model with state-of-the-art models. Experimental results show that our model outperforms the current state-of-the-art methods on two independent datasets.


Introduction
Drug discovery is a complicated process, which has the risks of long research cycles, high costs, and low success rates. It takes billions of dollars and more than ten years to develop a new drug from development to approval. 1,2 The effective prediction of drug-target binding affinity (DTA) is one of the signicant issues in drug discovery. [3][4][5] Drugs are usually represented as a string obtained from the simplied molecular-input lineentry system (SMILES) 6 or represented by a molecule graph with atoms as nodes and chemical bonds as edges. Targets (or proteins) are sequences of amino acids. Binding affinity indicates the strength of drug-target pair interaction. Through binding, drugs can have a positive or negative inuence on functions carried out by proteins, affecting the disease conditions. 7 By understanding drug-target binding affinity, it is possible to nd out candidate drugs that can inhibit the target/protein and benet many other bioinformatics applications. 8,9 Early computational attempts were focused on biologically intuitive methods, such as ligand-similarity based approaches and docking simulations. 10 Ligand-similarity based methods predict interactions by comparing a new ligand to known ligands of proteins. However, ligand-similarity methods perform poorly when the number of known ligands is insuf-cient. Docking simulation methods require the 3D structures of the target proteins hence becoming inapplicable when there are numerous proteins with unavailable 3D structures. 11 In the past few years, some research has begun predicting DTAs from a network perspective. [12][13][14] However, the prediction qualities of network-based approaches are strongly limited by available linked information. Thus, these methods do not perform very well on association predictions for new drugs or targets with scarce linked information. Additionally, some useful information, such as drug and target feature information, cannot be fully utilized to improve prediction accuracy for these methods.
With the development of articial intelligence, deep learning approaches for DTA prediction have become popular and can be categorized into two main groups according to the input data: sequence-based and graph-based methods. The sequencebased methods learned the representations from sequential data, which are SMILES sequences of drugs and amino acid sequences of proteins. The graph-based methods represented drugs as molecular graphs, which learned the representations from molecular graphs and amino acid sequences of proteins. Although deep learning models show excellent performance improvement in DTA prediction, two main challenges remain to study. First, these methods consider either SMILES sequences or molecular graphs, which failed to capture comprehensive representations of drugs. A SMILES sequence can offer the following features as a representation: (i) ionic groups and atomic groups are represented in the canonical way, which avoids confusion with their surrounding atomic groups. For instance, ammonium is denoted as [NH4+] rather than HHNHH; (ii) some specially dened symbols are used to preserve chemical properties such as chemical valence, isotopes, etc. However, merely taking drugs with sophisticated internal connectivity as simple sequential data lacks sufficient interpretable and expressive capabilities. Molecular graph brings two unique benets as compared to SMILES sequence: (i) molecular graph can capture the spatial connectivity of different atoms, especially for star structures and ring structures (e.g., alkyl and benzene ring); (ii) chemical molecular bonds are well preserved, which might inuence the molecular properties. For instance, carbon dioxide has divalent bonds between carbon and oxygen. However, similar to sequence modeling in SMILES, simply using molecular graphs to model molecules cannot enable methods to comprehensively learn molecular representations. It is difficult to capture information on some specic molecular properties, such as atoms' chirality, using molecular graphs. Second, most existing methods utilized convolutional neural networks (CNNs) to learn low-dimensional feature representations of proteins from the sequence of amino acids, which ignored the long-distance relationships in the protein sequences.
To overcome the mentioned challenges of current methods for DTA prediction, we propose a novel triple-channel model, named Graph-Sequence Attention and Transformer for Predicting Drug-Target Affinity (GSATDTA), to predict the binding affinity between drugs and targets. Recently, Guo et al. 15 proposed that integrating the capabilities of both molecular graphs and SMILES sequences can further enhance molecule representation expressive power. Enlightened by this work, we use a graph neural network to learn the graph representation from molecular graphs and a BiGRU to learn the SMILES representation from SMILES sequences. Then, we propose a graph-sequence attention mechanism to capture signicant information from both the SMILES sequence and molecular graph. For the protein representations, we replace CNN with an efficient transformer to learn the representation of the protein, which can capture the long-distance relationships in the sequence of amino acids. 16 The main contributions of this paper are summarized as follows: We leveraged both the graph and sequence information and proposed a graph-sequence attention mechanism to learn effective drug representations for DTA prediction.
We utilized an efficient transformer to learn the representation of the protein, which can capture the long-distance relationships in the sequence of amino acids.
We conduct extensive experiments on two benchmark datasets to investigate the performance of our proposed model. The experimental results show that our proposed model achieves the best performance in the drug-target binding affinity prediction task.

Simulation methods
Most previous works have focused on simulation-based methods (i.e., molecular docking and descriptors). For example, Li et al. 17 proposed a docking method based on random forest (RF). The RF model was also adopted in KronRLS, 18 which uses the similarity score obtained by the Kronecker product of the similarity matrix to improve the predictive performance. To alleviate the limitation of linear dependence in KronRLS, a gradient boosting method is proposed in SimBoost 19 to construct the similarity between drugs and targets. While classical methods have shown rational performance in DTA prediction, they are usually computationally expensive or rely on external expert knowledge or the 3D structure of the target/protein.

Sequence-based methods
The sequence-based methods learned the representations from sequential data, which are SMILES sequences of drugs and amino acid sequences of proteins. For example, DeepDTA 20 uses the 1D representation of the drug and protein, and uses convolutional neural networks (CNNs) to learn representations from the raw protein sequences and SMILES strings. Then, they combined these representations to feed into a fully connected layer to predict the drug-target affinity scores. Similarly, WideDTA 21 also relies only on the 1D representation, but different from DeepDTA, the SMILES and protein sequence were represented as words (instead of characters) that correspond to an eight-character sequence and a three-residual sequence, respectively. In addition, WideDTA utilized the ligand maximum common substructure (LMCS) 22 of drugs and motifs and domains of proteins (PDM), 23 which formed a fourbranch architecture together with the ligand SMILES and protein sequence branches. The WideDTA model is an extension of DeepDTA, which also used CNN to learn the representations of drugs and proteins. GANsDTA 24 utilized a generative adversarial networks (GAN) to learn benecial patterns within labeled and unlabeled sequences and used convolutional regression to forecast binding affinity scores. MATT_DTI 3 also utilizes the 1D representation, it proposed a relation-aware selfattention block to model the relative position between atoms in drugs, considering the correlation between atoms. As for the protein, it utilizes three convolutional layers as the feature extractor, followed by a max pooling layer. Then, a multi-head attention block is built to model the similarity of drug-target pairs as the interaction information for DTA prediction.

Graph-based methods
The graph-based methods represented drugs as molecular graphs, and learned the representations from molecular graphs and amino acid sequences of proteins. Tsubaki et al. 25 proposed the application of the graph neural network (GNN) to DTA (or compound-protein interaction, CPI) prediction. In their model, the chemical structures of drugs (provided in SMILES notation) are represented as graphs. Therefore, they propose the use of GNN and CNN, which can learn low-dimensional real-valued vector representations of molecular graphs and protein sequences. Similarly, Gao et al. 26 utilized the GNN for drug representation, whereas the protein descriptors were obtained using long short-term memory (LSTM). GraphDTA 5 also introduced graph representation to take advantage of the topological structure information of the molecular graph. GraphDTA used a three-layer GCN as an alternative for drug representation while utilizing the CNN to learn protein representation as in DeepDTA. Among the research on deep learning for drug discovery, DeepGS 27 is the most relevant to our work. DeepGS considers both the molecular graphs and SMILES sequences of drugs, however, they directly contacted the two representations, which failed to capture comprehensive information about drugs. Furthermore, DeepGS and other deep learning methods utilized CNN to learn local representations of protein sequences, which ignored the long-distance relationships in the protein sequences.
Compared with these deep learning models, we designed a graph-sequence attention mechanism, which can capture signicant information from both SMILES sequences and molecular graphs. As for the protein, we replace CNN with an efficient transformer to extract the long-distance relationships in the sequence of amino acids.

Materials and methods
We regard DTA prediction as a regression task to predict the binding affinity value between a drug-target pair. We denote the SMILES sequence set as D ¼ {D i } jDj i¼1 and the protein sequence set as P ¼ {P j } jPj j¼1 . We use RDKit to convert the SMILES set D into a molecular graph set denotes a molecular graph. Specically, a node on a molecular graph represents an atom, and an edge represents a chemical bond between two atoms. Formally, the problem of drug-target binding affinity prediction is dened as follows.
Given a SMILES set D and a protein set P, and their inter-

Overview
We proposed a triple-channel model, called GSATDTA; the overall architecture of the model is shown in Fig. 1. It takes the symbolic sequences of the target/protein and drug as inputs, as well as the molecular structure of the drug, and outputs the binding affinity of the drug to the target. For the drug representations, we use a graph neural network to learn the topological structure information from molecular graphs and BiGRU to learn contextual information about drugs represented by SMILES sequences. Then, we utilized the graph-sequence attention mechanism to capture signicant information from both the SMILES sequence and molecular graph. For the protein representations, we employed an efficient transformer to capture the long-distance relationships in the sequence of amino acids. Thus, we obtained the target representation and the fusion drug representation. The connection of two representations is inputted into several dense layers and ends with Fig. 1 Illustration of the proposed GSATDTA. We leveraged both the graph and sequence information and utilized the graph-sequence attention mechanism to learn effective drug representations. For the protein, we employed an efficient transformer to learn the representation of the protein sequence. Finally, the two representations were concatenated and passed through several dense fully connected layers to estimate the output as the drug-target affinity value. a regression layer to predict the drug-target binding affinity value. Next, we will present the details of our model.

Representation learning of drug
3.2.1 Representation learning of sequence. Simplied Molecular-Input Line-Entry System (SMILES) is a non-unique representation that encodes the molecular graph into a string of ASCII characters. For example, the SMILES "CN(C)CC]CC(] O)." for a drug in Fig. 1 is a sequence of atoms and chemical bonds. SMILES rules can cover atoms, ions, chemical bonds, valences, and chemical reactions, which can accurately express branched, cyclic, tetrahedral, aromatic, and chiral structures, as well as the expression of isomers and isotopes.
Most previous studies adopted one-hot encoding to encode the symbols in the SMILES sequence. However, one-hot encoding ignores the contextual value of the symbol, and thus cannot reveal the functionality of the symbol within the context. 28,29 To address this problem, we utilized Smi2Vec, 30 a method similar to Word2Vec, 31 to encode the tokens in the SMILES sequences. According to DeepGS, 27 we xed maximum lengths of 100 for SMILES sequences. We cut the SMILES sequence if the length of the SMILES sequence is longer than 100. Otherwise, we use zero-padding at the end of the SMILES sequence. In a typical case, a xed-length SMILES string, for example, C, is partitioned into individual atoms or symbols. It is then mapped to atoms by seeking out each atom embedded from a pretrained dictionary, and if not in the dictionary, it is obtained from randomly generated values. 32,33 Then, atom embedding vectors are aggregated to form the nal embedding matrix. Enlightened by the gate function in GRU, 34 we applied the three layers technique of BiGRU to the generated matrix to obtain a latent representation of the drug, which allows us to model the local chemical context. Finally, we obtain the SMILES representation S i˛R N through the max-pooling layer and the fully connected layer, where N is the output dimensional of the fully connected layer.

Representation learning of graph.
A vital indication for the estimation of DTA is to effectively exploit molecular structure information to reveal the interconnections between atoms in the drug. 5,25 To achieve this, we transformed the SMILES sequence into the molecule graph through the RDKit. Graph Isomorphic Network (GIN) 35 has shown its superiority for modeling graph representation in many studies and supposedly achieves maximum discriminative power among graph neural networks. Our model consists of ve GIN layers, each GIN layer is followed by a batch normalization layer, activated by a ReLU function. Specically, GIN uses a multi-layer perceptron (MLP) model to update the node features as formula (1): where 3 is either a learnable parameter or xed scalar, X j i˛R F is the feature vector of node j in the molecular graph i, and N(j) is the set of nodes adjacent to node j. Finally, a global max-pooling layer is added to aggregate the entire graph representation H iR N .

3.2.3
Graph-sequence attention. The attention mechanisms allow the network to focus on the most relevant parts of the input and have been proven to be useful for various tasks. 4,36 To capture signicant information from both the SMILES sequence and molecular graph, we designed an attention mechanism, called Graph-Sequence Attention. Specically, given the SMILES representation S i˛R N and the graph representation H i˛R N , we transform S i and H i into the vectors d S i and d H i through formula (2) for feature extraction and attention modeling: where W 1˛R N and W 2˛R N are trainable parameters, and b represents the bias vector. Then, we combine the output d S i with d H i through a dimensional-wise fusion gate F. F is accomplished by the sigmoid activation function to encode two parts of the representation: where W G˛R N and W S˛R N are trainable parameters of the fusion gate. Finally, the nal vector representation output of a specic drug is generated through F: where is the element-wise product.

Representation learning of protein
The protein sequence is a string of ASCII characters, which represents amino acids. Mathematically, a protein sequence is expressed as P j ¼ {p 1 , p 2 , ., p i , .}, where p i˛N * and the length of P j depends on the proteins. We x the length of the input protein sequence as L p to ensure the same size of inputs. According to the token embedding and position embedding in the transformer, 16 where T p˛RL p ÂM . 3.3.1 Efficient transformer. Most protein sequences are long sequences, so simple CNN cannot capture the longdistance relationships in protein sequences well. Therefore, current DTA prediction models are limited due to that CNN cannot capture the long-distance relationships in protein sequences. The transformer can alleviate this limitation, which offers a more exible mechanism to model the longdistance relationships. The multi-head self-attention (MSA) in the transformer encoder layer has been modeled and integrated to improve the model's ability to learn long-distance relationships. Multi-head attention can jointly attend to information from the extracted features at different sequence positions. The self-attention layer takes three inputs, the keys, K, the values, V, and the queries, Q, and calculates the attention as follows: where ffiffiffiffiffi ffi d K p is a scaling factor depending on the layer size. However, the memory and computation for multi-head selfattention in the traditional transformer scale quadratically with spatial or embedding dimensions (i.e., the number of channels), causing vast overheads for training and inference. Thus, we replace the multi-head self-attention with Efficient Multihead Self-Attention (EMSA). 37 The architecture of the efficient multi-head self-attention is shown in Fig. 2.
Similar to MSA, EMSA rst adopts a set of projections to obtain query Q. To compress memory, the 2D input T p˛RL p ÂM is reshaped to a 3D one along the spatial dimension (i.e., c T p˛RMÂhÂw ) and then fed to a depth-wise convolution operation to reduce the height and width dimension by a factor s. To make it simple, s is an adaptive set by the feature map size or the stage number. The kernel size, stride and padding are s + 1, s, and s/2, respectively. The new token map aer spatial reduction c T p˛RMÂh=sÂw=s is then reshaped to a 2D one, i.e., c T p˛RL 0 p ÂM , L 0 p ¼ h=s Â w=s. Then c T p is fed to two sets of projection to get key K and value V. Aer that, we adopt eqn (7) to compute the attention function on query Q, K and value V.
Here, conv($) is a standard 1 Â 1 convolutional operation, which models the interactions among different heads. As a result, the attention function of each head can depend on all of the keys and queries. However, this will impair the ability of MSA to jointly attend to information from different representation subsets at different positions. To restore this diversity ability, we add an Instance Normalization 38 (i.e., IN($)) for the dot product matrix (aer somax). Then, the output values of each head are concatenated and linearly projected to form the nal output. Finally, the protein representation is obtained through the max-pooling layer and the fully connected layer, which we denote as T j .

Drug-target binding affinity prediction
In this paper, we treat the drug-target binding affinity prediction task as a regression task. With the representation learned from the previous sections, we can integrate all the information about the drug and target to predict the binding affinity value. Firstly, we concatenated the drug representation M d and the protein representation T j . Secondly, we feed it into three dense fully connected layers to predict the binding affinity value. Besides, we use ReLU as the activation function for increasing the nonlinear relationship. Given the set of drug-target pairs and the ground-truth labels, we use the mean squared error (MSE) as the loss function.

Datasets
Following previous works, we employ two widely used datasets dedicated to DTA prediction: Davis: 20  For both datasets, we use the same training/testing data ratio as MATT_DTA, 3 DeepDTA, 20 and GraphDTA 5 in our experiments, making the comparison as fair as possible. That is, 80% of the data is used for training and the remaining 20% is used for testing the model. For the same purpose, we use the same evaluation metrics as MATT_DTA, GraphDTA, and DeepDTA for evaluating model performance: the mean squared error (MSE, the smaller the better), r m 2 (the larger the better), and the concordance index (CI, the larger the better). Table 1 summarizes the details of the Davis and Kiba datasets.

Evaluation metrics
MSE is a common metric to measure the difference between the predicted value and the real value: whereŷ i is the predicted value, y i is the true value, and N is the number of drug-target pairs. CI is used to measure whether the predicted binding affinity values of two random drug-target pairs were predicted in the same order as their true values: where b i is the prediction value for the larger affinity d i , b j is the prediction value for the smaller affinity d j , and h is the step function, as shown in eqn (10), and Z is a normalization constant that equals the number of drug-target pairs with different binding affinity values.
The metric r m 2 is used to evaluate the external prediction performance of QSAR (Quantitative Structure-Activity Relationship) models. A model is acceptable if and only if r m 2 ¼ r 2 Â ð1 À ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi r 2 À r 0 2 p Þ, where r 2 and r 0 2 are the squared correlation coefficient values between the observed and predicted values with and without intercept, respectively.

Baseline methods
We compare our model with the following state-of-the-art models: KronRLS: 18 this baseline formulates the problem of learning a prediction function f as nding a minimizer of the following objective function: (11) where N is the number of drug-target pairs, kfk k 2 is the norm of f, which is related to the kernel function k, and l > 0 is a regularization hyper-parameter dened by the user. SimBoost: 19 this baseline is a gradient-boosting machinebased method that constructs features of drug, target, and drug-target pairs. These features are fed into a supervised learning method named gradient boosting regression trees, which is derived from the gradient boosting machine model. Using a gradient regression tree, for a given drug-target pair d t i , the binding affinity scoreŷ i is calculated as follows: where M represents the number of regression trees, f m is a regression tree and F represents the space of all possible trees. The objective function with regularization in the regression tree set is described in the following form: 13) where N is the number of drug-target pairs, l is the loss function, y i is the true value,ŷ i is the predicted value, and a is a tuning parameter that controls the complexity of the model. DeepDTA: 20 this baseline trains two 3-layer CNNs using label/one-hot encoding to encode drug and protein sequences for DTA prediction. The CNN model contains two independent CNN blocks that capture features from SMILES sequences and protein sequences, respectively. The drug and target representations are concatenated and passed to a fully connected layer for DTA prediction.
WideDTA: 21 this baseline represents SMILES strings and protein sequences as word sequences and represents the corresponding drugs and proteins through the most common subsequence. In particular, drugs are described by the most common subsequences as Ligand Maximum Common Substructures (LMCS); proteins are represented by the most conserved subsequences that are Protein Domain proles or Motifs (PDM) retrieved from the PROSTE database. WideDTA contains four independent CNN blocks that learn features from SMILES sequences, LMCS, protein sequences, and PDM. The drug and target representations are all concatenated and passed to a fully connected layer for DTA prediction.
GANsDTA: 24 this baseline proposes a semi-supervised generative adversarial networks (GANs)-based method to predict binding affinity. This method comprises two types of networks, two partial GANs for the feature extraction from the raw protein sequences and SMILES strings separately, and a regression network using convolutional neural networks for prediction.
DeepGS: 27 DeepGS considers both the molecular graphs and SMILES sequences of drugs, and uses BiGRU to extract the local chemical context of SMILES sequences and GAT to capture the topological structure of molecular graphs. For the protein sequences, the CNN module is utilized to learn protein representations from the sequences of amino acids. Then, the representations of drugs and targets are concatenated and passed to a fully connected layer for DTA prediction.
GraphDTA: 5 this baseline converts drugs from SMILES sequences to molecular graphs. GraphDTA consists of two separate modules, a GNN module for modeling molecular graphs to obtain drug representations, and a CNN module for modeling protein sequences to obtain target representations. The drug and target representations are concatenated and passed to a fully connected layer for DTA prediction.
MATT_DTI: 3 this baseline use SMILES sequences and protein sequences as inputs. Unlike DeepDTA, MATT_DTI proposes a relation-aware self-attention module to model SMILES sequences. The relative self-attention module can enhance the relative position information between atoms in a compound while considering the relationship between elements. Aer the drug and target representations are obtained, a multi-head attention mechanism is used to model the interaction of drug representations and protein representations for DTA prediction.

Results and discussion
To examine the competitiveness of our proposed model, we compared the model with the current state-of-the-art models on the DTA prediction task. Tables 2 and 3 show the performance of different models on the Davis and Kiba datasets based on MSE, CI, and r m 2 metrics. As shown in Tables 2 and 3, those classical methods such as SimBoost 19 perform worse than deep learning methods. This is because classical methods rely heavily on manually annotated features provided by a domain expert and drug-target similarity matrices. In comparison, deep learning methods can capture more hidden features by using the automatic feature extraction ability of CNN and GNN. First, we consider a few recent textual representation approaches such as: DeepDTA, 20 WideDTA, 21 GANsDTA, 24 and MATT_DTI. 3 Among these approaches, MATT_DTI achieved the best results in terms of CI and MSE. MATT_DTI achieved CI of 0.890, and MSE of 0.229 on the Davis dataset; also, on the Kiba dataset, it achieved CI of 0.889 and MSE of 0.150. This explains the effectiveness of the attention mechanism in learning drug information in the case of MATT_DTI.
Second, we also consider some graph network approaches, GraphDTA and DeepGS. These graph representation approaches can effectively capture topological relationships of drug molecules, which enable further performance improvement. Amongst them, the GraphDTA shows a higher CI value of 0.893 and a lower MSE of 0.229 on the Davis dataset; and on the Kiba dataset, GraphDTA achieved 0.889 in terms of CI and 0.147 in terms of MSE. From Tables 2 and 3, we can nd that although DeepGS considers both the molecular graphs and SMILES sequences of drugs, it performs worse than GraphDTA in terms of CI, MSE, and r m 2 . The reason is that DeepGS directly contacted the SMILES representation and graph representation, which failed to capture comprehensive information about drugs.
As shown in Tables 2 and 3 We observe that our model outperforms existing deep-learning methods on three measures, which can be explained due to two factors: (1) Compared with these models, we replaced CNN with an efficient transformer to learn the representation of the protein, which can capture the long-distance relationships in the sequence of amino acids.
(2) We adopted the GIN architecture to learn the structural information of the molecular graphs and employed BiGRU to obtain extra contextual information for the SMILES sequences. Then, we utilized the graph-sequence attention mechanism to capture signicant information from both SMILES representation and graph representation.

Ablation experiment
To further validate the effect of the different components in GSATDTA, we designed two variants: GSATDTA-a and GSATDTA-b. GSATDTA-a is mainly for investigating the graphsequence attention, while GSATDTA-b is used to demonstrate the effectiveness of the efficient transformer.
GSATDTA-a used BiGRU to learn the SMILES representation and GIN to learn the graph representation. Then, it directly contacted the two representations without graph-sequence attention. For the protein, GSATDTA-a utilized the efficient transformer to learn the protein representation.
GSATDTA-b used BiGRU to learn the SMILES representation and GIN to learn the graph representation. Then, it utilized the graph-sequence attention to fuse the SMILES representation and the graph representation. For the protein, GSATDTAb replaced the efficient transformer with CNN to learn the protein representation.
From Table 4, we can nd that in the DTA prediction task, the performance of GSATDTA-a is worse than GSATDTA. These results demonstrate that the graph-sequence attention applied in GSATDTA is benecial to learning a comprehensive representation of the drugs. Furthermore, we can also observe that GSATDTA-b performs worse than GSATDTA. The result of ablation experiments indicates that the efficient transformer is efficient and effective to learn a good representation of proteins. Further, in an attempt to verify our hypothesis about the effectiveness of BiGRU in capturing contextual information from input SMILES, we evaluated the performance of the proposed GSATDTA on the Davis dataset using different versions of RNNs as presented in Fig. 3. We nd that simple RNN attains the highest MSE value with 0.224, and there is 0.006 improvement achieved when using GRU. Also, BiLSTM achieved an extra improvement of 3.7%, while BiGRU obtained the lowest MSE value with 0.200, which outperformed the BiLSTM performance by 4.8%. This experiment demonstrates the effectiveness of using BiGRU for modeling the SMILES sequence input.
In the molecular property prediction task, Guo et al. 15 proposed that integrating the capabilities of both molecular graphs and SMILES sequences can further enhance the model performance. To verify this work in the DTA prediction task, we also conduct an ablation experiment to investigate whether integrating the capabilities of both molecular graphs and SMILES sequences can further enhance the model performance in the drug-target binding affinity prediction task. We designed another two variants: GSATDTA-G and GSATDTA-S. GSATDTA-G uses GIN to learn the graph representation while GSATDTA-S utilizes BiGRU to learn the SMILES representation. For the protein, these two variants both employ the efficient transformer to learn the protein representation. To make the comparison as fair as possible, we chose GSATDTA-a to compare with them. Furthermore, for the evaluation metrics of GSATDTA-G and GSATDTA-S, we use the same metrics as our proposed method, which are the MSE, r m 2 , and CI. Fig. 4 and 5 illustrate the predicted against measured (actual) binding affinity values for the Kiba dataset. A perfect model is expected to provide a p ¼ y line where predictions (p) are equal to the measured (y) values. From Fig. 4 and 5, we can observe that compared with GSATDTA-S and GSATDTA-G, GSATDTA-a is denser around the p ¼ y line. More specically, in regions ① and ④, GSATDTA-S performed better than GSATDTA-G, while in regions ②, ③, and ⑤, GSATDTA-G performed better than GSATDTA-S. For GSATDTA-a, only a few points are spread in these areas. From Fig. 4 and 5, we can also nd that the overall trend of GSATDTA-a is more similar to GSATDTA-G, but in regions ① and ④, GSATDTA-a is more similar to GSATDTA-S and performs better than GSATDTA-G. We believe that the topological structure information of the molecular graph is critical to the DTA prediction task, but the local context features in SMILES sequences can be used as supplemental information to predict drug-target binding affinity. In conclusion, the results of visualization indicate that integrating the capabilities of both molecular graphs and SMILES sequences indeed can further enhance the model performance in the drug-target binding affinity prediction task, which is consistent with the viewpoint of Guo et al. 15     6 and 7 show the visualization of predicted against measured (actual) binding affinity values for the Davis dataset. We can nd similar performance in Fig. 6 and 7. In region ①, GSATDTA-G performed better than GSATDTA-S, while in region ②, GSATDTA-S performed better than GSATDTA-G. GSATDTAa performed better than GSATDTA-G and GSATDTA-S in region ③ and there are only a few points spread in these areas for GSATDTA-a.
Furthermore, we also represent the prediction performance of our proposed model based on the predicted value and measured (actual) value ( Fig. 8 and 9).
In the Kiba dataset, we analyzed the samples with large errors on the test set and found that when the protein sequence length exceeds 1000, there will be large errors. This is because we xed the length of the protein sequence to 1000 according to DeepDTA 20 and DeepGS, 27 which leads to information loss when extracting protein features. There are 4086 samples in the test set with protein sequence lengths longer than 1000. We calculated that the MSE of these samples is 0.174, which is higher than the overall MSE (0.126) of the test set. The MSE of the remaining 15 623 samples in the test set is 0.120, which is lower than the overall MSE of the test set. Therefore, when the sequence length of the protein is longer than 1000, our model suffers from large prediction errors.
We can nd the same performance in the Davis dataset. In the Davis test set, there are 1333 samples with protein sequence lengths longer than 1000. We calculate that the MSE of these samples is 0.214, which is higher than the overall MSE (0.200) of the test set. The MSE of the remaining 3677 samples in the test set is 0.192, which is lower than the overall MSE of the test set.

Conclusions
In this paper, we proposed a novel model based on selfattention, called GSATDTA, to predict the binding affinity between drugs and targets. We leveraged both the graph and sequence information and proposed a graph-sequence attention mechanism to learn effective drug representations for DTA prediction. Furthermore, we utilized an efficient transformer to learn the representation of proteins, which can capture the long-distance relationships in the sequence of amino acids. Extensive experimental results show that our model outperforms the state-of-the-art models in terms of MSE, CI, and r m 2 on two independent datasets. Since the transformer has achieved great success in various tasks, in the future, we will    consider investigating the graph transformer to learn the representations of the drug to predict drug-target binding affinity.

Conflicts of interest
There are no conicts to declare.