{'fraction_non_alphanumeric': 0.030080921916705503, 'fraction_numerical': 0.010041096470787611, 'mean_word_length': 5.036700767263427, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Few-Shot Learning of Accurate Folding Landscape for Protein Structure Prediction

Data-driven predictive methods which can efficiently and accurately transform protein sequences into biologically active structures are highly valuable for scientific research and therapeutical development. Determining accurate folding landscape using co-evolutionary information is fundamental to the success of modern protein structure prediction methods. As the state of the art, AlphaFold2 has dramatically raised the accuracy without performing explicit co-evolutionary analysis. Nevertheless, its performance still shows strong dependence on available sequence homologs. We investigated the cause of such dependence and presented EvoGen, a meta generative model, to remedy the underperformance of AlphaFold2 for poor MSA targets. EvoGen allows us to manipulate the folding landscape either by denoising the searched MSA or by generating virtual MSA, and helps AlphaFold2 fold accurately in low-data regime or even achieve encouraging performance with single-sequence predictions. Being able to make accurate predictions with few-shot MSA not only generalizes AlphaFold2 better for orphan sequences, but also democratizes its use for high-throughput applications. Besides, EvoGen combined with AlphaFold2 yields a probabilistic structure generation method which could explore alternative conformations of protein sequences, and the task-aware differentiable algorithm for sequence generation will benefit other related tasks including protein design.


Introduction
According to the energy landscape theory [1,2,3], proteins live on a rugged landscape consisting of numerous local minima corresponding to various metastable conformations. Among all these local minima, there may exist a dominant one, and the corresponding conformation is often easier to be experimentally determined than others and attributed to biological activity. Strictly speaking, protein structure(s) is a probabilistic distribution conditioned on its sequence and the interacting contexts such as ions and solvents rather than a deterministic function of its sequence alone. Physics-based protein models, like force-field molecular dynamics [4], can in principle sample all function-relevant metastable conformations of a protein [5], but in practice this is almost impossible due to limited time budget. More direct and fast methods which can fold proteins into relatively stable structures are highly desired, giving rise to various protein structure prediction (PSP) algorithms.
In data-driven PSP, the structure label assigned to a sequence (usually determined by experiments) is often unique, thus, the mathematics is simplified to finding a function which can map a sequence of amino acids (denoted by s) into a set of 3D coordinates (denoted by x throughout the paper). In most folding engines [6], this mapping is formulated as an implicit function x = argminx U θ (x; s), where U θ (x; s) is a learnable energy (or score) function defining the folding landscape of the sequence. Such energy-based implicit modeling is also popular in other machine learning scenarios where iterative optimizations or multiscale complexities are involved [7,8]. The overall objective is usually dissected into two sub-tasks: i) defining a proper energy landscape, over which the global minimizer corresponds to the ground truth (i.e., target structure observed by experiments); ii) defining an efficient minimizer searcher (or "optimizer") which could find the target minimum across the energy landscape. The illustration of traditional PSP workflow is illustrated in Fig. 1a where an ideal convex folding landscape is assumed. However, due to the large dimensionality, we often need to deal with non-convex folding landscape as illustrated in Fig. 1b. Therefore, initial guesses and the choice of optimizer become crucial, and, for example, a template or reference structure may serve as a good initial guess and simplify the folding process [9]. On the other hand, constructing a smooth folding landscape based on available information is even more important for data-driven PSP. Many approaches have been developed to build the folding landscape according to physics priors or observed data. For example, direct coupling analysis (DCA) [10,11] over multiple sequence alignment (MSA) and associated approaches (e.g., GREMLIN [12,13]) aim to construct the folding landscape (or folding restraints) for the target sequence according to evolutionary homologs, whereas AlphaFold1 [14] and related work [15,16] further showed that deep learning in combination with traditional coevolutionary analysis can increase the accuracy and smoothness of the folding landscape. Historically, construction of folding landscape and finding the minimizer of the landscape were usually performed separately. With the advent of deep learning, some efforts tried to fulfill these two tasks in an end-to-end manner such as Differentiable Simulator [17], RoseTTAFold [18] and AlphaFold2 (AF2) [19]. Nevertheless, how to solve the above-mentioned two tasks still evidently influenced the design of end-to-end models. For example, the Evoformer module in AF2 mainly serves to learn the folding landscape based on evolutionary information [19,20], while the equivariant Structure Module plays a role of meta optimizer [21,22] given that recent studies have connected gradient descent with equivariant networks [23].
Different from most PSP models including AF1, AF2 takes raw MSA as input and does not require DCA or other statistics of MSA. In theory, a sufficient MSA depth which is essential for DCA is no longer necessary for AF2, and single-sequence PSP via AF2 is also possible. Although AlphaFold2 has raised up the baseline of PSP accuracy, however, it suffers significant drop of performance when MSA is limited [19]. Therefore, it evokes us to rethink the cause of such gap between the state-of-theart model and the well-known Anfinsen's hypothesis [24] that the protein structure can be determined by its sequence. Besides, it is also appealing to investigate whether it is possible to close this gap, considering that once PSP can be made accurate with few or even without MSA, it would democratize PSP models without deploying resource-consuming and ever-growing sequence database, speed up the exploration of the protein universe, and help us better handle "orphan sequences". Besides, how to explore various conformations based on data-driven PSP models is still an open question [25], and it is to our interest that whether we can arrive at different local minima or metastable conformations by manipulating the folding landscape of end-to-end PSP models like AF2.

Shed light on the black-box folding landscape of AlphaFold2
In traditional PSP models, the depth, coverage, and diversity of MSA are known to influence the quality of DCA and the accuracy of the resulting folding restraints. However, since AF2 uses raw MSA as input without any explicit DCA-like feature extraction, little is known about how MSA influences the folding landscape of AF2, although we do know that the performance of AF2 drops dramatically as available MSA decreases [19]. Therefore, we designed an experiment to examine how AF2 responds to varied MSAs. We first selected a query sequence in CASP14 dataset [26], and randomly sub-sampled a certain number of MSA from the full MSA pool. We then fed these random MSA samples to AF2 and examined whether AF2 would produce varied structures for the same target. 2 may lead to different (sub)optimal structures. Reasonalbe initial guess according to some reference or template structure x ref could help find the correct minimum. (c), TMScores of AF2 predictions for T1032 in CASP14 with randomly sub-sampled MSA at varied MSA depth. 48 independent trials were performed for each MSA depth, and the result was reported in box plot. (d), Distribution of the inter-quartile range (IQR) between 1/4 quartile (Q1) and 3/4 quartile (Q3) for all sequences in CASP14 test set at varied MSA depth. For each sequence we performed the same random trials as in panel c and collected its IQR of TMScores. The IQRs of all applicable targets at a given MSA depth were compiled in the form of box plot. Unless stated otherwise, the hat lines of box plots correspond to (Q1-1.5IQR) and (Q3+1.5IQR) respectively. (e), Illustration of the proposed mechanism of how AF2 extracts folding landscape according to evolutionary information. Each sequence homolog in MSA has its own sequence-dependent folding landscape U θ (x; s), and AF2 manages to integrate these individual landscapes into a new one U θ (x; {s}), where most of the noisy and disturbing local minima are averaged out and keeps only the target minimum commonly shared by all homologue sequences.
As Fig. 1c shows, when we raised MSA depth up to 64 or more, AF2 is able to consistently produce a "converged" structure regardless of the randomness in the input MSA. However, intriguingly, the diversity of produced structures significantly increases as the number of MSA samples decreases. We further investigated this phenomenon using a larger test set containing 84 CASP14 targets ( Fig.  1d; see Appendix for more details). It was confirmed that, with a sufficient large MSA depth, AF2 tends to fold the protein into converged structures regardless of the randomness in MSA samples; In contrast, given a small number of MSA, the folded structures produced by AF2 are particularly sensitive to the identity of the selected MSA.
This observation can be reasonably explained by the well-known landscape theory of protein folding [2,3]. Consider that each MSA sequence associated to a query has its own folding landscape consisting of different local minima (corresponding to different metastable conformations), some of the local minima are unique to an individual sequence, but one or a few local minima, such as the target structure of the query sequence, may be shared by most of the homologue sequences. A possible mechanism of how AF2 constructs folding landscape from MSA is that Evoformer manages to integrate the individual folding landscape of MSA sequences (Fig. 1e): Most of the noisy local minima which are specific to few certain sequences are whitened or averaged out; On the other hand, those local minima shared by most of homologue sequences are eventually kept, resulting in a reasonable folding landscape.
It follows from this proposed mechanism that, when the number of available MSA becomes smaller, the remains of the disturbing folding minima are more likely to impact the integrated folding landscape and make it more rugged. It also implies some principles of how to select MSA for PSP consistent with practical rules-of-thumb: i) MSA should not be too distant from the query so that they share a common target minimum; ii) MSA should not be too similar to the query so that their individual local minima are independent; iii) MSA should be diverse (i.e., should not be too similar to each other) so that noisy local minima can be averaged out, rendering the integrated landscape relatively smooth. But in practice, we still do not know how to "optimize" MSA according to these rules, because MSA selection is not differentiable w.r.t. downstream goals. Even in the so-called end-to-end models like AF2, MSA selection is decoupled with the training of model parameters. Besides, the "exploration-exploitation" dilemma was also observed in our experiments, where some converged structures produced at a high MSA depth are sub-optimal compared to certain structures occasionally produced with a smaller subset of MSA. It is thus desired to develop a data-driven MSA optimization strategy which can automatically create useful MSA patterns to guide models like AF2 to fold better or explore alternative conformations.

Deep probabilistic learning of MSA
The most common probabilistic model for MSA is arguably the Potts model [27,28], which describes the distribution of amino acids across MSA as a Markov random field, and is widely used in DCA and GREMLIN. Although Potts model belongs to the family of generative models, it has several limitations: Potts model ignores any coupling between amino acids in MSA higher than the second order. Optimization of likelihood of Potts model involves the calculation of intractable partition function, so in practice specific gradient approximation methods like pseudo-likelihood are used. Worse still, a Potts model is only meaningful for a single set of MSA based on which the model is inferenced, and cannot directly transfer to another set of MSA. Recently, deep learning based approach was proposed to allow amortized optimization of Potts models across different MSAs [29]. However, it still assumes a pairwise coupling form and approximates the gradient using pseudolikelihood. In order to optimize MSA, we developed a parametric probabilistic model for MSA, which overcomes the shortcomings of Potts model. Given a dataset (denoted by D) containing many sets of MSA, each MSA set m in D is defined for a center query sequence Q m , and m ≡ {S i m } i=1,. . . ,Nm contains N m aligned sequences with length of L m amino acids (note that S 1 m ≡ Q m ). Similar to Potts model, our goal is to construct a statistical model p θ (θ denotes optimizable model parameters) which maximizes the likelihood of the observed MSA, Note that unlike Potts model, the likelihood function Eq. (1) is written for the full-order joint distribution without any limited-order approximation. It is also different from masked language models like BERT [30] where only marginal distribution of masked amino acids in MSA is modeled [31]. There has been work leveraging Eq. (1) for a specific set of MSA [32]. However, since we hope our model can be transferable to different MSAs, we reformulate Eq. (1) using conditional probability in the form of Eq. (2), To allow the model using more available conditional information, we further relaxed Eq. (2) into a multi-sequence conditional likelihood as conventionally used in meta generative learning [33,34,35], where or generalizability of MSA patterns. Researchers already know how to infer structural information from MSA patterns (i.e., p(x|{S})), and we in turn hypothesize that MSA patterns are subjected to a common implicit rule which can be learned by a transferable model. According to Bayesian theorem, p({S}|x) ∝ p(x|{S})/p(x) , this "hidden rule" is probably the 3D structure of proteins, and useful MSA patterns are indeed evidence of the protein structure during evolution. Following this reasoning, similar 3D structures may lead to similar MSA patterns, and our aim is to build a model which learns the relation from structure to MSA, and decodes meaningful structure-related MSA patterns.
Inspired by the success of autoregressive variational models [36,37], we derived an autoregressive variational lower bound to tame the intractable likelihood [38]. The variational bound becomes tight when the expressivity of the probabilistic model is sufficiently large, so we adopt a deep neural network to model the conditional distribution in Eq. (3), and we named this deep neural network model as EvoGen. In the design of EvoGen, we sticked to two basic principles: "relativity" and "hierarchy". Given context sequences as input, the model is designated to learn the relative difference between targets and contexts in order to ease the training. Besides, it is well-known that sequences which are different at amino-acid level may be very similar in property or structure, therefore, it is reasonable to employ multiple feature spaces for amino-acid embedding in order to compare the relative difference between protein sequences. We thus introduced hierarchical feature spaces (or latent spaces) in EvoGen inspired by advanced deep generative models [37,39]. Moreover, since we aim to model the full-length dependence between amino acids in a sequence, we do not use sequential autoregressive models which decode a sequence character by character in a given order. Instead, EvoGen is able to generate whole amino acids of a sequence simultaneously through a diffusion-like generation process (Fig. 2a). Overall, the inference and generation of EvoGen are fulfilled by a U-shaped [40] model which is widely adopted in modern diffusion probabilistic models [41].

Unsupervised MSA calibration remolds the folding landscape
In order to optimize MSA and smooth the folding landscape, we expect our model to enhance the folding-relevant signals while whiten the disturbing ones in manually searched MSA. Therefore, we first curated a MSA dataset [42] which has good coverage and sufficient depth (see Appendix for more details). We then trained EvoGen on this dataset according to Eq. (3) in an unsupervised manner. Provided that most of the MSA in the dataset contain folding-relevant signals, such an encoder-decoder scheme, as commonly adopted for denoising settings [43], can teach the model to reconstruct the common folding-relevant patterns while suppress the noisy or disturbing signals. During inference, EvoGen is able to transform a set of input MSA into a set of less noisy MSA, and we term this transform as MSA calibration. Note that the number of MSA does not change during MSA calibration. Since the input and output of EvoGen are both MSA, it can be directly plugged into the inference workflow of AF2 without any fine-tuning or modifications. From this respect, Eq.
(3) and EvoGen can be regarded as a new type of pretraining approach for PSP.
We then benchmarked EvoGen on a curated CASP14 test set (see Appendix for more details). To test whether our model can help smooth the folding landscape under poor MSA settings, we limited the number of MSA accessible by AF2 to be no more than 128 and ran all inference without templates. We made fair comparisons by running AF2 inference with and without MSA calibration when the same set of MSA were fed as input. Note that we turned off any settings which could cause nondeterministic effects during AF2 inference (see Appendix for more details). However, since EvoGen is a probabilistic model, it can yield varied output MSA even if the input MSA is constant. We ranked the output structures according to the predicted confidence (i.e., averaged per-residue plDDT) and reported the most confident structure (called "first"). By convention, we also reported the best scored structure (called "best") assuming the ground true score is known. Fig. 3a shows that, given the same set of MSA, without any fine-tuning that requires structure labels, EvoGen could improve AF2 predictions over CASP14 targets.
Consistent with previously proposed mechanism (Section 2.1), we found that the improved performance is more significant for targets of small MSA depths. Therefore, it is appealing to check whether MSA calibration can help AF2 predictions over "hard targets" which naturally lack homologue sequences. Such targets are often termed as "orphan sequences", and we curated a "poor MSA" test set which consists of single protein chains with known PDB structures but with less than 30 available MSAs (see Appendix for more details). We then benchmarked EvoGen on this dataset following the same procedure as described for CASP14. From Fig. 3b it can be concluded that EvoGen is able to effectively improve AF2 predictions for targets of which the available MSA is noisy or insufficient.
We further compared the output structures of AF2 with and without MSA calibration. EvoGen can help AF2 predict the correct loop conformations using calibrated MSAs (Fig. 3c). Calibrated MSA can also help AF2 form correct secondary structures, for instance, from wrong helices to correct sheets as shown in Fig. 3d. In some cases (Fig. 3e), EvoGen even rescued AF2 from failed predictions by restoring the overall structures. We noticed that the restored parts of structure mostly correspond to sub-sequences with limited MSA coverage and regions of additional flexibility. This finding indicates that EvoGen may help smooth the folding landscape by promoting the folding-relevant signals which are noisy or less pronounced in the original MSA, thus reinforcing the target folding minimum.
Given a probabilistic model, we can now treat the "sequence-to-structure" problem from a probabilistic view (Eq. (4)) rather than the commonly used deterministic mapping: where f represents EvoGen, z is a random vector coming from the standard normal distribution, and g is a (deterministic) function such as AF2 which maps MSA to a 3D structure. Equation (4) has a dramatic advantage that, by feeding different random Gaussian noises z to EvoGen, we can generate different structural conformations for a same sequence via AF2 even all the input sequences s (including MSA) are fixed. As introduced previously, during benchmark we generated several different structures (or decoys) for each target sequence by simply varying the random noises. For some target sequences, EvoGen plus AF2 led to different structural ensembles. Except high-scored structures when compared to the ground truth, there may also exist some highly confident (according to plDDT) but much lower-scored decoys. We hypothesized that such kind of highly confident decoys are likely to be alternative conformations of the target protein. Fig. 4 provides the visualization of exemplary cases encountered during benchmark.
As an intriguing case, 3VNE (PDB code), which corresponds to protein VP24, is one of the eight proteins encoded by ebolaviruses [44]. VP24 is known to contribute to immune suppression and can bind host transcription factor STAT1 [45]. EvoGen plus AF2 successfully generated an ensemble of predicted structures (Ensemble 1 in Fig. 4a) consistent with 3VNE which records the crystal structure of VP24 in an isolated monomer form. However, by virtue of probabilistic structure generation (Eq. (4)), we also observed another ensemble of predicted structures (Ensemble 2 in Fig. 4a), which shows subtle but clear differences when superimposed with Ensemble 1. Particularly, there is a relative rotation and displacement of α3-4 and β5-7 in Ensemble2 compared against the monomer structure (as indicated by the red arrows in the superposition of Fig. 4a). Intriguingly, according to the deuterium exchange experiment [45],β5-7 happens to reside near the hypothetic PPI interface between VP24 and STAT1. On the other hand, α3-4 also shows dramatic changes in deuterium exchange rate after binding to STAT1 [45], indicating the occurrence of local conformational changes. Therefore, structures in Ensemble2 which were not recorded in any PDB entry may possibly correspond to an alternative conformation of VP24 in the form of complex with other proteins like STAT1. In a similar case, given the same set of MSA but feeding varied random noises to EvoGen, we also obtained two remarkably different structure ensembles via AF2 for target 2X5T (PDB code). As shown in Fig. 4b, structures belonging to Ensemble 1 are highly similar to each other, and they also align well with the structure label in PDB, where two identical chains form an homodimer interfaced at the helix bundle [46] (highlighted by the red box in Fig. 4b). In contrast, another ensemble of highly confident structures (Ensemble 2 in Fig. 4b) was also observed. Structures in Ensemble 2 also align well with each other, and the main difference between these structures and those in Ensemble 1 lies in the overturn of the helix bundle (see the superposition in Fig. 4b) and the breakage of the PPI interface observed in the crystal structure. Considered that Ensemble 1 corresponds to conformations when the protein aggregate to dimers, Ensemble 2 may represent an alternative conformation when the protein takes an isolated or other complex form.

Generative MSA augmentation stabilizes few-shot folding
Given that calibrated MSA could improve the folding landscape, it naturally invokes the following question: Since searching and aligning natural sequences may cause disturbing noises in MSA, is it possible to directly "create" virtual MSA patterns which could form smooth folding landscape? This problem mirrors the well-known problem of inverse protein design, where a sequence needs to be generated to stabilize or fit a specific protein structure. The subtle difference here is that we aim at generating MSA for a given sequence based on which a PSP model can easily fold the protein into the targets structure. Similar to the inverse protein design problem, the generated MSA should also stabilize the targe structure and create a relatively smooth folding landscape. Therefore, following the approach of actor-critic learning [47], we further trained EvoGen under the guide of AF2: EvoGen plays the role of actor or generator and randomly creates MSAs, whereas AF2 plays the role of critic to judge whether the generated MSA can help fold the target sequence into the correct structure. Thanks to the differentiability of virtual MSA generated by EvoGen, we can directly optimize parameters of EvoGen through back-propagation and the chain rule: We first computed the gradient of the loss functions of AF2 with respect to MSA, and multiplies the gradient of MSA with respect to the parameters of EvoGen. Therefore, EvoGen allows us to create MSA or protein sequences for the downstream objective in an end-to-end differentiable manner and optimize the model using any efficient first-order optimizers. To fine-tune EvoGen for PSP, we include supervised structural losses in AF2 (including FAPE and torsional angle losses) in our optimization objective. Besides, it is known that some sequences that are very different may mutually share a similar fold, therefore, we regularize EvoGen to generate sequences not far away from naturally existing MSA patterns. This goal can be easily achieved by adding Eq. (3) into the final optimization objective, which encourages EvoGen to "imitate" how nature evolves protein sequences.
After fine-tuning EvoGen with the help of AF2, we tested whether EvoGen could generate MSAs which form reasonable folding landscape. We performed the benchmark following conventional procedures in few-shot learning: We gradually reduced the number of available MSA to AF2 and EvoGen, and compared the performance of AF2 with and without MSA generated by EvoGen. Indeed, EvoGen hereby is equivalent to an approach of data augmentation for AF2, so we term this procedure as MSA augmentation in order to mark its difference with MSA calibration. In MSA augmentation, EvoGen takes a small set of MSA as input but outputs a new and larger set of MSA.
We first benchmarked MSA augmentation on CASP14 test set. Consistent with the original paper of AF2, we found that by decreasing the number of input MSA, the performance of AF2 gradually drops (Fig. 5a). In contrast, with the augmentation of MSA provided by EvoGen, the overall prediction quality is relatively stabilized, and the drop of performance is effectively soothed. Particularly noteworthy, with only tens of MSA, EvoGen could keep the accuracy of AF2 near the same level as the full-MSA inference workflow. We also benchmarked the performance of EvoGen over the CAMEO test set (see more details in Appendix), and plotted the result in Fig. 5b, which led to the same conclusion as above We further performed MSA augmentation for the poor-MSA test set and checked whether it can help improve the prediction of AF2 over "hard targets". Since sequences in this dataset have limited number of searched MSA, they can call trouble for AF2 inference. As shown in the scatter plot (Fig.  5c), given the same set of available searched MSA, many targets failed for original AF2 inference can now be accurately predicted with MSA augmentation. This experiments show that EvoGen could help AF2 achieve the state-of-the-art performance on naturally poor-MSA sequences.
We noticed that not all the targets we tested can be improved with MSA augmentation, so we analyzed and compared the structural characteristics of proteins where MSA augmentation can or cannot improve predictions. We first collected all the test sequences in the three datasets (CASP14, CAMEO and poor-MSA), and extracted two subsets: The "improved set" contains sequences for which MSA augmentation corrects the originally mis-folded predictions (AF2 TMScore<0.5 whereas EvoGen+AF2 TMScore>0.55); while the "underperformed set" consists of sequences for which MSA augmentation causes underperformance (∆TMScore<-0.05). The compositions of the secondary structures were calculated for these two sets (shown in cyan-colored columns in Fig. 5d). The overall structural compositions of the two sets are quite similar except that α-helix is less redundant in the improved set (upper panel in Fig. 5d) compared to the underperformed set (lower panel in Fig. 5d). We also computed how much fraction of each secondary structure element was correctly predicted with and without MSA augmentation. For targets in improved set, all types of secondary structural elements were more accurately predicted with MSA augmentation. Particularly, some rare structures like 3 10 -helix and 5-helix were almost completely failed by AF2 but were successfully restored with MSA augmentation. In contrast, we did not observe significant performance gap (except β-bridge) for underperformed targets in terms of secondary structures. This comparison indicates that MSA augmentation could generally improve the structural predictions of AF2 with limited risk of causing underperformance. We visualized several exemplary targets for which MSA augmentation significantly improved the prediction. In Fig. 5e we compared the structures predicted by AF2 with or without MSA augmentation for three proteins with limited number of MSA. It can be seen that MSA augmentation can not only help improve structures of coil-abundant proteins like 4BFH (PDB code), but also correct the folding chirality of the anti-freeze protein 1Z2F (PDB code). Besides, the overall structure of a viral DNA polymerase 1T6L (PDB code) is also rescued by EvoGen.

How far are we from ideal single-sequence protein structure prediction?
Finally it comes to a widely concerned question: Whether the structure of a protein can be accurately predicted merely by its sequence without using any other information like MSA or templates? Anfinsen's experiments showed that amino-acid sequence determines the stable structure of a protein.
However, it is well-known that protein structures are only marginally stable and often dynamic [48], and many proteins have more than one metastable conformation, so the mapping from sequence to structure may not be deterministic. Consequently, single-sequence PSP can be rather complicated and challenging due to the high-dimensional nature of the folding landscape (see Fig. 1 for illustration), and most modern PSP models rely on additional information other than query sequence to reduce the complexity of such a high-dimensional non-convex optimization problem. Nevertheless, EvoGen provides a possible approach which formally enables MSA-based models to perform single-sequence PSP. Specifically, if no MSA information except for the query sequence is provided to EvoGen, the model can also generate MSA in a zero-shot manner, and the generated MSA can be fed into downstream PSP models such as AF2, leading to a non-deterministic single-sequence inference workflow.
We first conducted experiments using zero-shot MSA generation (or equivalently, single-sequence PSP) via EvoGen in combination with AF2 over CASP14 and CAMEO test sets. Fig. 6a shows that, given merely the query sequence AF2 cannot predict the correct fold (defined as TMScore>0.5 by convention) for most of the targets in both test sets. However, if we provided AF2 with the MSA created by EvoGen through zero-shot generation, the overall quality of predictions was significantly improved, and more than a half of originally mis-folded structures were predicted with correct fold. This encouraging result promoted us to further test whether the zero-shot MSA generation plus AF2 works for natural "orphan sequences", thus, we performed the same experiment over the poor-MSA dataset and provided only the query sequence to EvoGen as conditional information. From Fig. 4b we can see that zero-shot MSA generation via EvoGen also significantly improved AF2 accuracy over these hard targets.
Since single-sequence PSP is not well-defined for proteins which exhibit multiple dynamically competing conformations [25], it is appealing to check whether EvoGen overfits the structure labels which often collapse multiple conformations into a specific one due to experimental conditions. We first divided the poor-MSA dataset according to the nature of the proteins into various categories, for example, whether the protein is natural or artificially designed, whether the protein is toxin or viral, and whether the protein structure is determined by X-ray or NMR. Intriguingly, we found that de novo proteins, viral proteins and toxins are specifically enriched in the poor-MSA dataset, indicating that these proteins naturally lack homologue sequences or co-evolutionary information. For each category of proteins, we computed how many AF2 predictions fall into correct fold (TMScore>0.5) with or without zero-shot MSA generation (Fig. 6c). We first observed that EvoGen can effectively help AF2 improve the quality of predicted structures for natural proteins. Particularly for viral proteins and toxins, the gains in performance are significant. Besides, EvoGen could help improve the quality of single-sequence PSP by a large margin for sequences whose structures are determined through Xray. In contrast, only limited improvement was observed for sequences whose structures are determined through NMR. This finding suggests that EvoGen can help improve AF2 predictions for sequences which only have one dominant stable structure, but does not overfit for sequences which exhibit dynamic structures as those determined via NMR.
Intriguingly, we find that single-sequence AF2 is able to predict the correct fold for most of the artificially designed proteins even without the help of EvoGen, although EvoGen still helps raise the quality of predictions to a higher level. This finding can be reasonably explained by the fact that, compared to natural proteins, most of de novo proteins are hyper-stable because the sequence itself is deliberately designed to stabilize the structure. The folding landscape of such hyper-stable protein is usually much smoother and there only exists one dominant minimum across the energy landscape. Since MSA information for de novo designed proteins is usually limited due to their artificial nature, single-sequence structure prediction is often needed for protein designs. We thus further benchmarked zero-shot EvoGen plus AF2 on a more commonly used de novo protein dataset [49,50]. As Fig. 6d shows, although the original AF2 already outperforms existing methods like RaptorX [49] in single-sequence PSP setting, AF2 still fails for some targets and yields wrong folds (TMScore<0.5). With the help of zero-shot MSA generation, all the targets can be predicted into correct folds and the overall performance reaches state of the art. in Fig. 6e, we visualized two de novo targets where the original single-sequence AF2 predictions failed. 6CZG is an artificially designed β-barrel protein, but without zero-shot MSA augmentation, AF2 cannot predict its correct topology and chirality. As the second case, 6W3F is an artificially designed enzyme-like protein containing an active pocket, but AF2 wrongly predicted its structure, especially that of the key active pocket, while EvoGen helps restore the correct fold and accurately predict the pocket structure.

Concluding Remarks
Fast and accurate PSP models are of great practical value because they make efficient exploration of protein sequence space possible, particularly in design applications [51,52]. The advent of AlphaFold2 has raised the bar for the accuracy of data-driven PSP, and it has been assisting researchers to expand the database of proteins since its birth [53]. AF2 is different from most of previous PSP models in that it is end-to-end, and that it inputs raw MSA features without any co-evolutionary analysis. In theory, AF2 is able to perform few-shot MSA or even zero-shot MSA (i.e., singlesequence) PSP. Without deploying resource-consuming and ever-growing sequence database, singlesequence PSP is quite appealing because it could democratize PSP models for large-scale and high-throughput applications, and dramatically speed up the exploration of the protein universe.
Besides, currently about 1/5 of all metagenomic protein sequences [54] and about 11% of eukaryotic and viral proteins [55] are estimated to be "orphan" which naturally lack sequence homologs. Dealing with these sequences requires PSP models to make accurate predictions with very limited evolutionary information. Unfortunately, the performance of AF2 is guaranteed only if the available MSA is sufficiently deep. Unlike DCA where large MSA depth is known as pre-requisite, how MSA influences AF2 and why shallow MSA harms the performance of AF2 is quite obscure. In this paper, we tried to open the black box of AF2, investigated the mechanism of how AF2 constructs folding landscape from provided evolutionary information, and proposed a simple but reasonable physical picture explaining the observed connections between MSA and AF2 performance.
Assuming that MSA is evidence of 3D structures, we designed a deep probabilistic model for MSAs called EvoGen. Different from Potts model, EvoGen is a full-order joint probabilistic model as well as a meta-generative model for MSA. It is designed to learn generalizable features across MSAs for different query sequences. We designed a specific U-shaped neural network architecture for EvoGen so that it is transferable to MSAs with varied depth and length. We also formulated a variational lower bound for the full-order joint likelihood so that the training and inference of EvoGen can be performed efficiently.
Based on EvoGen, two plug-in methods were developed to improve the accuracy of MSA-based PSP models like AF2 when dealing with poor MSA targets. On the one hand, serving as an unsupervised data-denoising strategy, MSA calibration is able to effectively whiten or denoise the manually searched MSA, and it can help remold the folding landscape thus help AF2 fold better. To achieve MSA calibration, EvoGen is trained only on the sequence database without any structure labels, and the downstream AF2 is not fine-tuned at all. On the other hand, MSA augmentation provides virtual generated MSA to downstream AF2 as data augmentation which can stabilize the prediction quality in low-MSA regime. Particularly, zero-shot MSA augmentation can help AF2 improve single-sequence predictions over hyper-stable proteins. To achieve MSA augmentation, we trained EvoGen under the guide of AF2 over a limited number of structure labels, and we can directly backpropagate the structural losses of AF2 to EvoGen owing to the differentiability of the generated MSA. Moreover, by functional compositions of the probabilistic EvoGen and a deterministic function like AF2, we obtained a new type of probabilistic end-to-end PSP algorithm, which could yield varied structures given a unique input. We showed that such probabilistic PSP algorithm could help explore alternative conformations of proteins, which can be crucial for drug discovery and therapeutics.
Generally speaking, EvoGen can be regarded as a model-agnostic pre-training strategy for proteinrelated tasks. Different from other language-modeling-based pre-training like BERT, we showed that EvoGen can even work well with limited change of downstream models. This is particularly beneficial in that unlike natural language processing, protein-related model is usually very large and complicated (e.g., AF2), so the fine-tuning is usually more difficult than pre-training itself. On the other hand, it is also possible to employ MSA generation instead of masked language modeling as the pre-training task and feed the latent representation learn to the downstream PSP model. As recent study shows, such pre-trained LM can help PSP models achieve reasonable single sequence predictions [56]. Besides, as a differentiable sequence generator, EvoGen can be trained straightforwardly according to downstream objectives, and generate optimized protein sequences efficiently without performing Monte Carlo or gradient descent. Therefore, we expect the model and algorithm behind EvoGen can also assist other sequence generation tasks like protein design [52] as well as MSA-based protein learning tasks such as functional annotation and mutation assessment [57] etc. in the future.

Code and Data Availability
Codes of EvoGen as well as associated model checkpoints will be released at MindSPONGE platform (https://gitee.com/mindspore/mindscience/tree/master/MindSPONGE). Test datasets and associated benchmark results will also be released.

Conflict of Interest
The authors declare no conflict of interest.

MSA trimming
Because all experiments in this paper were designed for low-data regime, we performed MSA trimming for CASP14 and CAMEO targets whenever MSA is abundant. Given a maximum MSA depth N max , the MSA trimming follows the same procedure as adopted for PSPD-Lite [42]. Specifically, for each query sequence whose MSA depth exceeding N max , we first filtered its MSA according to three primary rules: i) all MSA's with coverage less than 50% are removed; ii) all MSA's with >90% identity to query are removed; iii) all MSA's with <20% identity to query are removed. If MSA depth of the query still exceeds N max after filtering, we further selected representative MSA's via a heuristic strategy as follows: We initialized an MSA pool using the query sequence alone, then added to this pool a new MSA given that this candidate is of no more than 90% identity to all MSA's already in the pool, and that this candidate is closest to the query in terms of the Hamming's distance. This iterative selection stops when no more candidates can be accepted or the MSA pool is full (up to N max ). MSA trimming with N max = 128 was performed for CASP14 and CAMEO test sets.

Training sets
We curated two training sets for EvoGen. The "labeled set" contains both sequences and structural labels, while the "unlabeled set" is composed merely of sequences (and MSA) without structural labels. The labeled set consists of 447K filtered PDB structures extracted from PSPD-Lite with a date truncation before 2020-05-14. CASP14 and CAMEO test sets are naturally excluded from the training set. The unlabeled set further expands the labeled set by adding 648K filtered non-redundant sequences in UniRef50 [58] extracted from PSPD-Lite [42], and only the sequence information (i.e. MSA) is preserved whereas the structure labels are deprecated. For both training sets, MSA trimming is performed with a N max = 256 following the strategy described above. Additional filtering was performed after trimming: i) All entries with MSA depth smaller than 128 are removed (the poor MSA test set is thus excluded from the training set); ii) Any sequences in the de novo test set are manually removed; iii) Sequences or structural labels with length shorter than 20 amino acids are also removed. EvoGen was trained on the unlabeled set for MSA calibration, and fine-tuned on the labeled set for MSA augmentation.

AF2 inference settings
We conducted all the experiments without templates. AF2 model-3 released by DeepMind was chosen for inference and training in all experiments unless specified otherwise, because it slightly outperformed the other two template-free models (model-4 and model-5) on our benchmark dataset, and it is also recommended as default model by batch-mode ColabFold [59]. After MSA trimming, MSA subsampling is no longer performed during inference, except when we deliberately sub-sampled MSA for purposes (see Section I in Experiments & Results). We also turned off any other settings which could cause non-deterministic effects (e.g., BERT) during AF2 inference. Unless stated otherwise, AF2 inference was executed exclusively using a recommended number of three recycles.

Hyperformer
Hyperformer inherits the overall architecture of Evoformer [19] but exhibits several key differences (Fig. 2c). First of all, the original biased attention is replaced with hyper-attention inspired by Molecular CT [60], and the attention coefficient between a d-dimensional Query vector q i and Key vector k j is computed as where W ij and b ij are learnable parameters or activations of neural networks which are both functions of the relative positions (or pair activations) between i-th and j-th tokens. Similar to hyper-networks [61], W ij and b ij here represent learnable affine transform of the space basis and the offset of the resulting inner product, respectively. It can be seen that vanilla attention (or biased attention) is a special case of hyper-attention given an identity W ij and zero (or non-zero) b ij . The second difference lies in the embedding of relative positions. We adopted an approach similar to T5 model [62] and grouped |i − j| into discretized buckets according to log-scales [63]. This way of relative positional embedding not only expands the horizon of sequence models without inducing extra memory cost, but also equips the model with a hyperbolic view of distances as inductive biases. Thirdly, we added a new Query Conditioning Module into EvoGen encoder, which is a neural network that mixes context activations with query activations in order to help the model learn relative differences between MSA and the query sequence. Lastly, similar to AlphaFold-Multimer [64], we changed the order of the "outer product mean" operation to the beginning of each Hyperformer block, allowing the single update and pair update to be executed in parallel and separately.

Latent module
Latent module is designed to summarize the statistics of MSA features. Latent modules in encoder are responsible for summarizing contextual sequences into data-dependent priors, while latent modules in decoders are responsible for estimating the posteriors for target sequences. Considered the overall symmetry between encoder and decoder (Fig. 2a), and the principle of "relativity", we employed twin (or Siamese) matching networks to learn the relevant statistics (Fig. 2d). Another neural network called Sampling Module, draws random samples according to learned posteriors with Gaussian noises via re-parametrization trick [38].

Model hyperparameters
EvoGen is composed of a pair of relatively symmetric encoder and decoder similar to U-Net [40], each consisting of 12 Hyperformer blocks. According to the principle of "hierarchy", we adopted 3 Latent Module blocks between the encoder and decoder (Fig. 2a), with increasing latent dimensions during encoding (or equivalently, decreasing dimensions during decoding).