{'fraction_non_alphanumeric': 0.028342659569628316, 'fraction_numerical': 0.008188890405350075, 'mean_word_length': 4.965264586160108, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 5, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MusiteDeep: a deep-learning based webserver for protein post-translational modification site prediction and visualization

Abstract MusiteDeep is an online resource providing a deep-learning framework for protein post-translational modification (PTM) site prediction and visualization. The predictor only uses protein sequences as input and no complex features are needed, which results in a real-time prediction for a large number of proteins. It takes less than three minutes to predict for 1000 sequences per PTM type. The output is presented at the amino acid level for the user-selected PTM types. The framework has been benchmarked and has demonstrated competitive performance in PTM site predictions by other researchers. In this webserver, we updated the previous framework by utilizing more advanced ensemble techniques, and providing prediction and visualization for multiple PTMs simultaneously for users to analyze potential PTM cross-talks directly. Besides prediction, users can interactively review the predicted PTM sites in the context of known PTM annotations and protein 3D structures through homology-based search. In addition, the server maintains a local database providing pre-processed PTM annotations from Uniport/Swiss-Prot for users to download. This database will be updated every three months. The MusiteDeep server is available at https://www.musite.net. The stand-alone tools for locally using MusiteDeep are available at https://github.com/duolinwang/MusiteDeep_web.


INTRODUCTION
Protein post-translational modifications (PTMs) generally refer to the additions of various functional groups to amino acid residues (1). PTM is a key mechanism to increase proteomic diversities and plays an important role in the regu-lation of protein functions (2). Therefore, identifying and understanding PTMs are critical in the studies of biology and diseases. To date, as the accumulation of PTM experimental data, dozens of bioinformatics tools have been developed for PTM site prediction, which provide a fast and low-cost approach in contrast to experimental methods. Many of these tools apply a machine-learning algorithm and provide a prediction for a particular PTM type. Here, we briefly review several representative tools. Musite (3) was proposed by our group and applies a support vector machine (SVM) for phosphorylation site prediction, using the K nearest neighbor (KNN) score, disorder scores and amino acid frequencies as features. GlycoEP (4) used SVM for N-, O-and C-linked glycosylation site prediction. Besides raw sequence features, GlycoEP extracted amino acid composition (AAC) profiles, position-specific scoring matrix (PSSM) profiles, secondary structures, and surface accessibilities as features. GPS-PAIL (5) is a tool to predict HAT-specific lysine acetylation sites, which used their previously proposed GPS2.2 algorithm (6) as well as features from an amino acid substitution matrix (e.g. BLO-SUM62) and protein-protein interaction. GPS-SUMO (7) is another GPS algorithm-based tool to predict the sumoylation sites. MePred-RF (8) applied a random forest algorithm with a complex sequence-based feature selection technique to predict methylarginine and methyllysine sites. RF-Hydroxysite (9) applied a random forest algorithm for hydroxylysine and hydroxyproline site prediction that combined information from physicochemical, structural, evolutionary and sequence-order features. Css-Palm4.0 (10) applied a clustering and scoring strategy for palmitoylation site prediction. UbiProber (11) was designed for ubiquitin site prediction that uses KNN, physicochemical property, and AAC features to train an SVM-based predictor. Deep-Phos recently applied densely connected convolutional neural network (CNN) blocks for phosphorylation site prediction (12). Because the dysregulation of PTM plays impor-tant roles in the development and progression of diseases (13)(14)(15), a number of databases were developed to annotate existing or predicted PTM sites with SNPs or diseases, such as dbPTM (16), PTMcode (17) and AWESOME (18). They also provide predictions by assembling third-party tools. For example, the recently developed AWESOME applied 20 available tools for different types of PTM site predictions, and the method described in this paper was included for phosphorylation site prediction.
Despite the availability of these tools and databases, the number of webservers that provide a general prediction for many different PTM types is quite low. PTM-ssMP (19) is one, but it relies on known modification profiles, which cannot make de-novo prediction outside the profiles. ModPred (20) provides a general webserver for 23 different modifications. However, it was published in 2014 and applies the logistic regression method, with a performance in need of improvement by more advanced machine learning methods. Most of the other existing predictive webservers were developed for a single type of PTM prediction, such as GlycoEP and MePred-RF. All these servers limit the number of submissions (mostly a single sequence), and none of them supports batch submission for any large-scale prediction, due to calculations of more complex features, such as PSSM, which takes significant computation time.
Here, we introduce a new webserver, MusiteDeep, to provide a general deep-learning framework for protein PTM site prediction and visualization. The method was first introduced in (21). To the best of our knowledge, MusiteDeep was the first deep-learning method in phosphorylation (one of the most studied PTMs) site prediction. It takes raw protein sequences as input and uses CNN (22) with a novel twodimensional (2D) attention mechanism. It has been widely benchmarked by other predictors (12,(23)(24)(25) and has always ranked in first or second place. It has also been adopted in several other services (18,(26)(27). Later, we upgraded the deep-learning framework with the capsule network (28) and extended it to predict for more PTM types, which showed superior performance for almost all the cases with small training samples (29). In this webserver, we updated the previous framework by combining the two previous networks and utilizing more advanced ensemble techniques, while simultaneously providing prediction and visualization for multiple PTMs. The server provides a real-time prediction for a large number of proteins by using only CPU without GPU resources. Users can interactively review predicted PTM sites in the context of known PTM annotations and protein 3D structures through a homology-based search. In addition, the server maintains a local database (updated every three months) to provide pre-processed PTM annotations from Uniport/Swiss-Prot for download. Comparing with the existing web services, MusiteDeep has some obvious advantages in accuracy, speed and scale, and it also provides some unique functions for analyzing prediction results.

Method overview
Deep-learning framework. The framework of MusiteDeep for protein PTM site prediction is shown in Figure 1. We treated the PTM site prediction problem as a binary classification problem, and for each type of PTM, we trained an independent predictor. In the training process, the framework accepts raw protein sequences as input, and then 33length residue fragments centered at the target sites were extracted and coded by the one-of-K coding method. We considered the 20 common amino acids and a hyphen character '-', which is used to pad the positions when the valid fragment length is less than 33. Therefore, each position is represented by a 21D vector, with value 1 at the index corresponding to the amino acid or the hyphen character, and a 0 at all other indexes; meanwhile, uncommon amino acids are filled with 0.05. The final output of the framework is a confidence score of the PTM prediction. In regard to the deep-learning architecture, we used the combination of the two previously proposed networks, i.e., MultiCNN (21) and CapsNet (29) as shown in Figure 1 right. The Multi-CNN model is constructed by three one-dimensional (1D) CNN layers, a two-dimensional (2D) attention layer, and two fully connected layers; the CapsNet is constructed by two 1D CNN layers, a PrimaryCaps layer and a PTMCaps layer. We trained both networks separately, and in the prediction procedure, a final prediction score is calculated by averaging the prediction scores obtained by the two independent networks. The architecture details can be found in the Supplementary Text S1.
Bootstrapping and weight averaging. Because nearly all PTM types have more negative samples than positive samples, to address the unbalanced issues during training, a bootstrapping technique was applied. As shown in Figure 1 left, the training fragments were partitioned into N subsets, each containing the same number of positive and negative fragments; here, N was determined by the integer part of the negative to positive ratio. The network was trained iteratively, and the number of iterations was set as N, or in practical use, an upper limit (30 by default). In each training iteration, one subset was used to train the network through the Adam stochastic optimization (30) based on mini-batches. After training for N iterations, one classifier was obtained. The early stopping strategy was applied in each iteration, and when the loss of a validation set did not decrease in some number of epochs (one forward and backward pass over the entire subset), the training procedure for that subset would stop. It has been demonstrated that an ensemble model created by averaging the weights from a continuous training procedure leads to wider optima and better generalization (31). Therefore, during the bootstrapping procedure, we applied the weight averaging strategy. We treated each iteration trained on one subset as one training cycle and we saved the weights generated from each iteration denoted as W i , and their loss on the validation set is denoted as L i . The final weight W of the classifier can be calculated by the weighted average of the weights from all the iterations as follows: To further improve the performance, we trained an ensemble of 10 × N C models using a nested cross-validation. The original training data was divided into 10 equal-sized subsets. Each time, nine subsets were used to train a model, and the remaining subset was used as validation to monitor its training process. Ten different training sets were used during the training. In the meantime, on the same training set, we repeated the bootstrapping procedure for N C times, and N C independent classifiers were obtained. N C is a hyper-parameter which can be determined by users, and N C = 2 was used in this work. The final prediction score was the average of the scores obtained from the 10 × N C ensemble models.
Transfer learning. For PTM types that have small training samples, we applied a transfer learning technique to further improve the performance. The lower layers of CNN serve as feature extraction layers, and the extracted features can be generalized to a different dataset. Consider the phosphotyrosine as an example. Because it is catalyzed by different kinase groups with phosphoserine or phosphothreonine, we and other researchers have trained separate models for them. However, they share some common features: For example, the phosphorylation always happened in the disorder region. Therefore, in this case, the transfer learning technique can be applied to use a larger dataset to capture the common features and to use the smaller dataset to expatiate its specified features. In particular, we trained a base network on the phosphoserine and phosphothreonine data; then, we used the pre-trained weights of the base network to initialize the weights for the phosphotyrosine predictor. Finally, we fine-tuned the weights of the predictor using the phosphotyrosine data.
Through the combination of the two architectures and the ensemble techniques, the overall performance has been improved, which is shown in the Supplementary Figure S1 by comparing the previous methods on the 10-fold crossvalidation datasets provided in (29).

Performance evaluation
Because different tools used different thresholds to present the prediction results, to avoid the effects of distinct thresh-old values, we used the area under the ROC curves (AUC) and the area under the precision-recall curves to evaluate the performance. Especially, when the negative data is very large, the precision-recall curves are more appropriate to evaluate the performance of a predictor.

MusiteDeep web server
Server inputs and outputs. The input of the server is protein sequences in the FASTA format. The server provides two options for input, paste mode and upload mode. For a small task with up to 10 sequences or 500 amino acids, users can paste their sequences to the input panel, and the job will start immediately and return the result in real time once submitted. For a larger-scale task, users can upload a FASTA file with up to 10 MB. In this mode, once the job is submitted, it will be placed in a queue for processing. One user can process up to five such jobs at the same time. The job can be accessed later by the provided URL or by checking the user's job history. It is important to notice that since we used the browser's local storage to remember the identity of a user, the job history only lists previous jobs that were successfully submitted by the user using the same browser; however, access through the URL has no such restriction. All the jobs will be saved on the server for 72 hours with up to 100MB per user. MusiteDeep does not have a complex feature calculation procedure so it can handle a large number of proteins by only CPU. It takes less than three minutes to predict for 1000 sequences per PTM type. Users can simultaneously select multiple models for prediction from the drop-down list. After a job is finished, the output can be visualized for each input sequence one by one, which can be retrieved by its sequence name or its index number. The output is shown in Figure 2 spond to their prediction confidence levels. Upon hovering the mouse on the predicted sites, the detailed information of the prediction will be shown. A user may adjust the prediction confidence threshold by using the slider to obtain more or fewer predicted sites. Besides interactive visualization, the results can be downloaded as plain text, with information of protein identifier, position, residue, PTMscore, and the predicted PTMs whose scores are higher than the user-defined or default cut-off. An example is shown in Supplementary Table S1. In the server, we also provide a REST API for this service along with a template Python program to demonstrate how to use the API.
View predicted PTM sites in the context of known PTM annotations. MusiteDeep provides a homology-based search against proteins in UniProtKB/Swiss-Prot, and presents known PTM annotations at the aligned positions, as shown in Figure 3. Protein accession identifier (ACCID), Blast sequence identity, and the known PTM annotations on homologous proteins are presented for each input sequence. Upon hovering the mouse over the colored sites, the specific annotation will be shown. Proteins can be accessed in UniProtKB/Swiss-Prot database by clicking their ACCIDs. In the server, we also provide a REST API for this service along with a template Python program to demonstrate how to use the API.
View predicted PTM sites in the context of protein 3D structure. MusiteDeep provides visualizations of the predicted PTM sites in 3D protein structures by integrating G2S (32), a tool to annotate genomic variants on protein structures, and an NGL viewer (33), which is a web application for molecular visualization. First, a query sequence with the predicted PTM sites is searched by G2S; then, its homologous proteins that have 3D structures in RCSB PDB (34) will be retrieved with the mapping between sequence positions and structural positions. The retrieved information is shown in Figure 4.
The query protein and its predicted PTMs can be viewed in the 3D structure context, as shown in Figure 5. The hover text shows the information of the predicted site, which contains its position on the query sequence, its amino acid types at the query sequence position and at the PDB structural position, and its predicted PTM types (in abbreviation). In this example, multiple PTMs are viewed at the same time, which gives the sense of 3D spatial locations of PTM sites. Users can propose several hypotheses accordingly, such as whether the predicted sites are physically near each other or whether they form a potential PTM cross-talk. i.e. whether these PTMs of multiple residues work together to determine a particular functional outcome. In the example of Figure  5, all the predicted PTM sites locate closely on the structure, which only takes place in the loop region. Since most PTMs tend to occur on the protein surface, we also provide the user an option to add the molecule surface by checking the 'Add surface' box.

Annotated PTM sequences from UniProtKB/Swiss-Prot.
The UniProtKB/Swiss-Prot database contains publicly available and expertly annotated protein sequences (35), including the PTM annotations. MusiteDeep maintains a local database of UniProtKB/Swiss-Prot, which provides preprocessed PTM annotations for users to download and is updated every 3 months.
Web server implementation. The webserver consists of a three-layer architecture of front-end, server-end, and business logic layers. The front-end of the server is implemented with JavaScript libraries, React and jQuery to provide an interactive user interface; the back-end is implemented with W144 Nucleic Acids Research, 2020, Vol. 48, Web Server issue    KOA, a next-generation web framework for node.js; the deep-learning framework was implemented in the business logic layer by Python. We used mongoDB for the server's database. The REST APIs are implemented by KOA and is available at the API section of the server. The standalone tools used to run several services on a local machine are available at GitHub (https://github.com/duolinwang/ MusiteDeep web) with detailed documentation.

Independent benchmarking
To demonstrate the performance of the PTM site prediction of MusiteDeep in practical use, we compared MusiteDeep with existing tools by using a timestamp-based dataset.
Because each type of PTM may have dozens of predictive tools, to compare with all of them is very challenging. Therefore, besides ModPred, which can cover all of our provided PTM types, we selected the most representative public available tools based on their performance and publication citation for each PTM comparison. The training data was constructed by extracting the protein sequences from UniProtKB/Swiss-Prot before 2010, and the timestampbased testing data was constructed by the newly released data after 2010. All the annotations generated by computational predictions were removed. For each PTM, all the residues annotated by UniProtKB/Swiss-Prot with the same type of PTM were treated as positive sites, while the residues with the same amino acids excluding the PTM annotations were regarded as the negative sites. The statistics of the data are shown in Supplementary Table S2. The performance of the whole testing data is evaluated by the area under the ROC curves and the area under the precisionrecall curves, as shown in Table 1. We also evaluate the performance on test subsets that have different levels of sequence similarities to the training data. The results are shown in Supplementary Figure S2, which shows that most prediction performances hold well with low sequence similarities.

CONCLUSIONS
In this paper, we present MusiteDeep, a web server for protein PTM site prediction and visualization. The method behind the server is a combination of our previously proposed two deep-learning models implemented with two ensemble techniques. Since the predictor does not require the calculation of complex features, the server is capable of providing real-time prediction and batch submission for large-scale protein sequences. The output is presented at the amino acid level for multiple PTMs at the same time. All the submitted jobs will be saved in the server for 72 h with up to 100MB for users to retrieve. Besides prediction, MusiteDeep provides facilities for users to interactively review the predicted PTM sites in the context of known PTM annotations and protein 3D structures. In addition, the server maintains a local database providing pre-processed PTM annotations from Uniport/Swiss-Prot for users to download. Compared with the existing web services, MusiteDeep has some obvious advantages in accuracy, speed and scale. Besides the web server, we provide user-friendly Web APIs to access several services through Python programs and stand-alone tools, which allow users to run MusiteDeep on local machines.
Some limitations of this work include only providing models for PTM types that have enough data in UniProtKB/Swiss-Prot and focusing exclusively on a general modification induced by adding functional groups to the side chain of intermediate residues. In our future work, we plan to extend the framework to more PTM types, including the peptide cleavage and N-terminal PTMs, such as N-terminal acetylation and proteolysis. We will also expand our training data to include more databases on PTMs. The challenge that remains is how to combine the data from different sources and how to control the potential errors. These challenges will be the topics of our future studies.

DATA AVAILABILITY
MusiteDeep is available as a web server at https:// www.musite.net. The stand-alone tools to run several MusiteDeep services on a local machine are available in the GitHub repository (https://github.com/duolinwang/ MusiteDeep web).

SUPPLEMENTARY DATA
Supplementary Data are available at NAR Online.