{'fraction_non_alphanumeric': 0.03274664750957854, 'fraction_numerical': 0.008441091954022989, 'mean_word_length': 4.778277412659979, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Pareto-optimal compositional energy-based model for sampling and optimization of protein sequences

Deep generative models have emerged as a popular machine learning-based approach for inverse design problems in the life sciences. However, these problems often require sampling new designs that satisfy multiple properties of interest in addition to learning the data distribution. This multi-objective optimization becomes more challenging when properties are independent or orthogonal to each other. In this work, we propose a Pareto-compositional energy-based model (pcEBM), a framework that uses multiple gradient descent for sampling new designs that adhere to various constraints in optimizing distinct properties. We demonstrate its ability to learn non-convex Pareto fronts and generate sequences that simultaneously satisfy multiple desired properties across a series of real-world antibody design tasks.


Introduction
Generative models have shown promise across various applications in the life sciences for generating chemically-and physically-plausible designs and in accelerating the process of scientific discovery. Part of this trend of adoption can be owed to the convincing examples created by generative adversarial networks (GANs) [13], variational autoencoders (VAEs) [23], energy-based models (EBMs) [8] and, more recently, diffusion models [35; 32; 5]. However, there are far fewer success stories in real-world industry applications [6; 15]. Some reasons include an overrepresenation of image datasets; a lack of evaluation protocols and metrics for synthetic data [37; 33]; challenges around controllable generation and training -for exmaple GANs; and challenges in generating samples that are different from have been seen during training [22; 2; 38]. Taken together, these serve to limit the applicability of generative modeling for real-world use cases.
Our work is motivated by this and, in particular, the need to accelerate the development and discovery of new molecules, namely therapeutic antibodies. Though several generative models have already been proposed for these purposes [10; 12; 31], generating new samples without guidance or control does not guarantee downstream success of the proposed molecules. In practice, each molecule has to satisfy multiple properties. For therapeutic antibodies, this could include properties such as binding affinity, polyreactivity, and viscosity [1; 20; 39]. Failure to account for these and other properties can lead to serious complications later on during scale-up, manufacturing, and clinical trials and the optimization of straightforward objectives does not necessarily translate into progress in the laboratory [28]. Figure 1: Example output of the proposed Pareto-optimal energy-based sampler (pcEBM) compared to a naive multi-objective sampling (cEBM). The green marker denotes the starting sequence; each point in the plot corresponds to a modified design of that starting point, aiming at improving affinity and nonspecificity (BV-ELISA or BV score). pcEBM introduces samples along a Pareto front, with candidates minimizing both objectives simultaneously, while candidates generated by the cEBM minimize only one objective.
Motivated by this challenge, we propose a new EBM for antibody design that simultaneously takes into account multiple properties that an antibody has to satisfy. The adherence to multiple properties is a challenge in itself, as it often involves optimizing multiple conflicting objectives. In antibodies, for instance, optimization of binding affinity alone may come at the expense of developability properties, parameters that govern the likelihood of success of a molecule during manufacturing and quality control .
In general, in optimizing multiple conflicting properties, it is often impossible to find a single sample that satisfies all of the objectives simultaneously [21]. We argue that, instead, one should aim to propose a set of diverse data points from the Pareto front [3] that correspond to different choices for various objective functions. Doing so enables a global perspective the optimal trade-off between objectives and can select a molecule according to their preference. To achieve Pareto optimality, we rely on recent advances in multi-objective optimization (MOO) [34; 27] as well as on compositional sampling with EBMs [8] to build a Pareto-optimal compositional EBM (pcEBM). Figure 1 exemplifies both why we need pcEBM and what we achieve with it when optimizing a design of an initial antibody sequence.
We first present the the necessary background on multi-objective optimization in subsection 2.1 and related work on compositional sampling with EBM in subsection 2.2 leading to pcEBM described in subsection 2.3. In section 3 we include empirical evaluation and discussion, before concluding in section 4.

Background and method
Problem setup. Antibodies are composed of two chains of amino acids (AA), which can be represented as sequences of characters. Each AA comes from an alphabet of 20 characters corresponding , typically of combined length L ∼ 250. We will denote the sequences as x = (x 1 , . . . , x L ), where x l ∈ {1, . . . , 20} corresponds to the AA type at position l. For each of those sequences, we measure m properties, such that f i : R L → R for all i ∈ 1, . . . , m.
Our goal is to generate new sequences x * with preferred values for each of the m properties. Since we cannot fully satisfy all of the properties/objectives simultaneously (they may be conflicting with each other), we are interested in finding samples which can not be further improved simultaneously for all the objectives, yielding the notion of Pareto optimality [4]. [4; 27]) For x 1 , x 2 ∈ R d , we say that x 1 is dominated by

Definition 2.1 (Pareto Optimality
A point x * is called globally Pareto optimal on R d iff it is not dominated by any other x ∈ R d . A point x * is called locally Pareto optimal iff there exists an open neighborhood N (x * ) of x * , such that x * is not dominated by any x ∈ N (x * ). The collection of globally (resp. locally) Pareto optimal points are called the global (resp. local) Pareto set. The collection of function values . . , f m (x * )) of all the Pareto points x * is called the Pareto front.

Multi-objective optimization
A simple approach to solving MOO is linear scalarization [21] based on a preference vector λ = [λ 1 , . . . , λ m ] from the probability simplex on [m], i.e., S = {λ : As we evaluate on λ from a grid of S, we hope that the corresponding x * λ approximates the Pareto front. The solutions obtained by linear scalarization lie in the convex envelope of the Pareto front, which renders the approach plausible for situations where the Pareto front is convex [27].
An alternative approach for non-convex Pareto fronts is multiple gradient descent (MGD) [7], which iteratively updates variable x along a direction that ensures that all objectives are decreased simultaneously and guides the optimization towards a Pareto improvement. Denote the gradient of the i-th objective as ∇ is a vector to be determined and η is a small step size. From a 1st-order Taylor expansion, )/η which represents the decreasing rate of f i when we update x along direction g(x). In MGD, g is chosen to maximize the slowest decreasing rate among all the objectives, that is: By doing so, g(x) is pushed to have positive inner products with all ∇ x f i (x). If the latter is impossible, will contain conflicting directions and we will get g(x) = 0 which terminates the procedure. By using Lagrangian duality, Désidéri [7] has shown that the above objective has the optimal value g( The above convex optimisation problem has a closed form solution when m = 2 and a fast algorithm for m > 2 [34; 19]. By construction, when the step size is small, MGD decreases monotonically all objectives simultaneously and will terminate at a local Pareto point.

Compositional energy-based models
Although EBMs have been around for many years [25; 24], recent interest in generative modeling has led to a resurgence of interest [8; 9]. EBMs learn to represent data by approximating an unnormalized probability distribution across data. They do so by learning an energy function E θ (x) parameterized by a neural network that maps each input x to a scalar real value interpreted as the energy and approximating the data distribution by the Boltzmann distribution under unit temperature: Training an EBM on a given data distribution is usually done by contrastive divergence [16]. Besides predicting the un-normalized data likelihood, EBMs can be used to sample new data points x from p θ (during both training and generation). Sampling is typically performed by some Markov-Chain Monte-Carlo (MCMC) procedure. We consider the common case of Langevin Diffusion (LD) where the samples are initialized from refinement/uniform random noise followed by iterative refinement: with k being the sampling step and η the step size.
The above sampling procedure can be easily extended to produce samples from the composition of p θ with other distributions of interest. Along this direction, Du et al. [9] considered the problem of sampling from m composed distributions, each modeling a different propertiey f i , an approach referred to as compositional EBM (cEBM). Du et al. proposed multiple variations for executing different logical operations over properties of interest, such as conjunction, negation, and disjunction. In the following, we focus on conjunction, which corresponds to combining individual EBMs as a product of experts: We also note that cEBM utilized the following sampling procedure: which amounts to applying LD w.r.t. E(x) = i E(x|f i ).

Pareto compositional sampling
Inspired by previous work on multi-objective optimization [7; 27] discussed in subsection 2.1, we here propose pcEBM, a method that samples from the Pareto front of m Boltzmann distributions of interest. Our approach entails utilizing multiple gradient descent to select a locally optimal direction that optimizes the slowest decreasing rate among all the objectives of the compositional EBMs: where α is a positive constant and the internal optimization problem is the same utilized in MGD and the external sampler is performing Langevin diffusion. Suppose that we are generating a new sequence x, either by starting from random noise or by improving an existing one. At each step k, if x k−1 is far from the Pareto front and the gradients ∇ x f i (x k−1 ) have non-negligible norm, Equation 4 will drive x k−1 closer to the Pareto front. On the other hand, when x * is close to the Pareto front and the gradients have nearly vanished, the noise dominates and x k−1 performs Brownian motion. As will be demonstrated experimentally, the addition of noise ω helps pcEBM to explore more efficiently the Pareto as compared to MGD, whereas the locally optimal determination of the optimization direction renders the optimization more effective than cEBM.
Experimental setup. We wish to generate plausible antibody sequences according to three key properties: (i) similarity to known human antibodies from the public database of observed antibodies [29] (Ab-like), (ii) binding affinity to antigens of interest (Aff ), and (iii) an experimental measure of nonspecificity -phenomena which can affect the dosing regimen, safety, and overall therapeutic profile of an antibody -which we refer to as the BV score [17] 1 . We use datasets from both public and proprietary sources to train three separate EBMs, each pertaining to a property of interest. Though alternative choices are also possible, we instantiate all models as sequence-based convolution networks [11]. More details on the architecture can be found in the Appendix. We use the same models for sampling new sequences with each of the baselines. The differences in implementation are as follows: MGD does not add any noise, cEBM uses Lanvegin dynamics but has uniform weights for all properties; ls-cEBM differs from cEBM in that it includes domain-informed weights; and finally pcEBM learns the optimal weights for each property per sequence. Each baseline relies on two main hyper-parameters, namely the step size η and the number of steps k.
To compare different approaches in terms of their ability to satisfy multiple properties, we adopt the Hyper-Volume (HV) metric from the multi-objective optimization literature [14] computed over the energy scores of each property model. In all experiments, we aim for minimizing the energy scores per property. Additionally, as a proxy for how well the sampled sequences capture the properties of interest we compute the minimal edit distance of each sampled sequence to the ones seen during the training of each property EBM (e dist ) [30]. More details about the computation of these metrics can be found in the Appendix.  Table 1: Hyper-Volume (HV) across multiple combinations of properties (as measured by energy score) across different step sizes η. Larger scores indicate bigger improvement. The best results for are typeset in boldface. Table 1 and Table 2 address Q.1. We find that EBMs trained on data possessing a single property of interest propose sequences that are similar to those sharing the same property, but are dissimilar to the sequences preferred by other objectives. In contrast, sequences generated by a multi-property EBMs have smaller distances to all properties compared to the single-property EBMs. Amongst the four considered multi-objective baselines, pcEBM generates sequences with greater hyper-volume and similarity to the training datasets, whereas MGD is a close second.
We surmise the difference between cEBM and pcEBM is owed to their speed of convergence. We confirm this in Figure 3 by plotting the energy score at each sampling step for both cEBM and pcEBM with a step size of η = 0.01. As observed, for the same starting sequence pcEBM converges faster. This result partially answers Q.3.  Table 2: Edit distance (mean with standard deviation in brackets) to sequences in the training set with property of interest. Results in bold are best per columns, smaller is better, considering e dist as a proxy for similarity to real data with the specific property. To further test our approach, we calculate HV across the three properties we aim to account for. Here, we consider the energy of each property EBM as proxy for the objective of interest 2 . The advantage of pcEBM is noticeable at smaller step sizes (see Table 1), while the variance in both edit distance and hyper-volume for different η is significantly smaller for pcEBM than others. Figure 2 depicts the Pareto front for two properties obtained from candidate sequences from each baseline. We find that the pcEBM front is broader than MGD-based approaches, which agres our supposition that employing LD leads to an improved coverage of the Pareto front as compared to pure gradient descent. Further, pcEBM and MGD fronts are closer to the origin that that obtained by direct LD, which demonstrates the advantage of the multi-properly optimization aspect of pcEBM, addressing Q.2 and Q.3.
Finally, for Q.4 we wish to explore the possibility for improving properties of already existing antibodies by using the variants of compositional EBMs. We focus our analysis on nonspecificity (BV) as we have an external SeqCNN classifier that can act as a surrogate pseudo-oracle. We start by screening the existing sequences to select only those with low BV scores, which we refer to as "seeds". We use sequences with poor experimental and predicted BV scores as seed sequences for generation and then reuse the nonspecificity pseudo-oracle to evaluate the newly proposed designs. Figure 5 shows the BV score improvement achieved by each baseline. We notice that the multi-objective methods manage to improve the worst BV scores from 0.27 to above 0.9, with MGD and pcEBM giving a slightly lower average score than cEBM (though all scores are comparably high).
The plots in Figure 4 provide further insight into the nonspecificity optimization process by examining a low-dimensional embedding of seed and optimized sequences. The embedding is obtained by using tSNE to project the last layer feature representations of the sequences obtained from the nonspecificity oracle, onto a 2D space. We color each point (sequence) by their BV score with blue indicating a good score and red a poor one. On the same plot, we overlay a seed sequence and its corresponding proposed designs. We see that both cEBM and pcEBM improve the probability  for a good nonspecificity score, and move the seed towards the section of the manifold covered by datapoints with good scores.