text:
Regularized feature selection in reinforcement learning
- 1k Downloads
- Citations
We introduce feature regularization during feature selection for value function approximation. Feature regularization introduces a prior into the selection process, improving function approximation accuracy and reducing overfitting. We show that the smoothness prior is effective in the incremental feature selection setting and present closed-form smoothness regularizers for the Fourier and RBF bases. We present two methods for feature regularization which extend the temporal difference orthogonal matching pursuit (OMP-TD) algorithm and demonstrate the effectiveness of the smoothness prior; smooth Tikhonov OMP-TD and smoothness scaled OMP-TD. We compare these methods against OMP-TD, regularized OMP-TD and least squares TD with random projections, across six benchmark domains using two different types of basis functions.
KeywordsFeature selection Reinforcement learning Function approximation Regularization Linear function approximation OMP-TD
1 Introduction
Representing value functions in large or continuous state spaces requires function approximation. Linear function approximation is one common approach, where the value function is represented as a weighted sum of basis functions. It is often beneficial to keep the number of basis functions low (Kolter and Ng 2009; Painter-Wakefield and Parr 2012); however, the right set is usually domain-specific.
Recent work in feature selection (Kolter and Ng 2009; Johns et al. 2010; Painter-Wakefield and Parr 2012; Petrik et al. 2010; Ghavamzadeh et al. 2011; Geist et al. 2012) and construction (Ghavamzadeh et al. 2010; Sun et al. 2011; Parr et al. 2007; Mahadevan and Liu 2010) reduces the size of the approximation set by selecting or constructing basis functions specifically for each domain, and are typically used in conjunction with a batch method (Bradtke and Barto 1996; Lagoudakis and Parr 2003; Ernst et al. 2005) to fit the weights. Of particular interest are greedy selection methods such as orthogonal matching pursuit TD (OMP-TD), and orthogonal matching pursuit with Bellman residual minimization (OMP-BRM) (Painter-Wakefield and Parr 2012), which aim to select basis functions predicted to cause the largest reduction in approximation error post selection. These methods use a set of samples collected from a fixed policy, and fit the basis function weights to approximate the value function for that policy. Unlike OMP-BRM which assigns weights to basis functions only once as they are selected, OMP-TD recalculates weights using a batch method after each selection. Appropriate batch method choices for OMP-TD include linear TD (Sutton 1988), least squares policy evaluation (LSPE) (Yu and Bertsekas 2009) and least squares TD (LSTD) (Bradtke and Barto 1996), of which LSTD is used in this paper. The result of either method is a tailored set of basis functions and their respective weights specific to the domain/task and current policy. We focus on learning the value function for a specific policy, however policy improvement can be achieved through policy iteration (Sutton and Barto 1998) by using the value function of the old policy (found using OMP-TD for example) to generate a new policy.
Although greedy selection of basis functions using current methods does lead to steadily improving approximations, the goal is to improve the approximation as rapidly as possible. OMP-TD often performs better in practice than OMP-BRM (Painter-Wakefield and Parr 2012), but even so can be slow to converge for large discount factors (Mahadevan and Liu 2010). In our experiments, values as low as \(\gamma =0.8\) reduced performance noticeably on some domains. This problem occurs because OMP-TD aims to select functions which will improve the representational power of the approximation set, such that with optimal fitting, the value function will move closer to the true value function at least as much as applying the Bellman operator would move it. Since the Bellman operator is a \(\gamma \)-contraction in the weighted \(L_2\) norm (Van Roy 1998), the reduction in \(L_2\) error depends heavily on \(\gamma \), with large \(\gamma \) leading to slow convergence.
With large \(\gamma \) however, value functions of environments with sparse reward functions and local transitions tend to be smooth. This is due to the Bellman equation, where the value of a state is equal to the discounted expected value of the next state and the expected reward. With sparse reward functions, the values between consecutive states change little, and if those consecutive states are close in distance sufficiently often, the value function as a whole can be expected to be smooth. Many domains based on physical systems exhibit spatially local transitions (Jonschkowski and Brock 2013), and sparse reward functions are common, making smoothness a reliably exploitable attribute of value functions.
We introduce feature regularization as a way to enhance performance during feature selection by giving preference to basis functions according to prior expectations of the value function. In standard regression, regularization is commonly used to avoid overfitting, where simpler (smoother) solutions—solutions with smaller weights—are preferred. This regularization is usually applied in the final fitting of weights with a predetermined set of basis functions. In the incremental feature selection setting, however, regularization can be applied to the selection of individual basis functions as they are selected. Such regularization could be used to reduce overfitting, or to introduce a prior, but unlike regularizing the fit of the entire function, weight-based regularization is inappropriate since incremental feature selection methods usually choose basis functions that have the largest weights (high impact on the value function).
Tikhonov regularization (Tikhonov 1963) is one way to incorporate domain knowledge such as value function smoothness into feature selection. We present the general form of this regularization and use it to form the smooth Tikhonov OMP-TD (STOMP-TD) algorithm, which uses a smoothness regularizer on the selection step of the OMP-TD algorithm. We also present an almost parameter free heuristic algorithm utilising smoothness; smoothness scaled OMP-TD (SSOMP-TD).
We show empirically that basis function smoothness is an effective regularizer in many reinforcement learning domains, and compare these two methods against methods with similar goals, regularized OMP-TD (ROMP-TD) (Needell and Vershynin 2009) and least squares TD with random projections (LSTD-RP) (Ghavamzadeh et al. 2010). We chose these methods because ROMP-TD focuses on the problem of overfitting in feature selection, and LSTD-RP deals with the problem of large feature spaces by projecting them onto a smaller space.
We empirically evaluate feature regularization across six benchmark domains using two different types of basis, showing that feature regularization can significantly lower prediction error. SSOMP-TD in particular performs reliably well with no increase in computational complexity and an easily chosen parameter.
2 Background
A reinforcement learning (RL) task is typically formalized as a Markov decision process (MDP) (Sutton and Barto 1998). An MDP is defined as a tuple, \(M = (S,A,P,R,\gamma )\), where S is the set of all possible states s, \(A = \{a_1,a_2,\ldots ,a_N\}\) is the set of all actions, P is the transition probability model where \(P(s,a,s')\) is the probability of moving from state s to state \(s'\) with action a, R is a real-valued reward function where \(r = R(s,a,s')\) is the reward for taking action a in state s and ending up in state \(s'\), and \(\gamma \in [0,1)\) is the discount factor (Sutton and Barto 1998). We define our state as a vector of real values, i.e., \(s = [x_1,\ldots ,x_d]\) where d is the number of dimensions, and each \(x_j\) is scaled to be between 0 and 1.
The goal is to learn a policy, \(\pi \), that maximizes the expected return (discounted sum of future rewards). \(\pi \) is a stochastic mapping of states to actions where the probability of choosing a particular action a in state s is given by \(\pi (s,a)\). Learning a policy can be achieved by learning a value function V if P is known, or an action-value function Q if P is not known. \(V^{\pi }(s) \in {\mathbb {R}}\) is the expected return an agent receives starting in state s by following policy \(\pi \) thereafter. The value function for a policy \(\pi \) satisfies the Bellman equation \(V^{\pi } = R + \gamma PV^{\pi }\), where \(V^*\) is the value function for an optimal policy. We only consider V since Q is similar.
Often, the number of states is too large to store individual values. A standard technique for representing value functions in such cases is value function approximation. Linear function approximation is a common choice. The approximate value function at a state s is represented by a linear sum of real-valued basis functions \(\phi _i(s) \in {\mathbb {R}}\) with associated weights \(w_i \in {\mathbb {R}}\): \(\bar{V}^{\pi }(s) = \sum _{i=1}^kw_i\phi _i(s) = \varvec{\phi }(s)\cdot \varvec{w}\), where \(\varvec{\phi }(s) = [\phi _1(s), \ldots , \phi _k(s)]^{T}\) and \(\varvec{w} = [w_1, \ldots , w_k]^{T}\). For multiple states \(\varvec{s} = [s_1,\ldots , s_q]^{T}\), the vector \(\bar{V}^{\pi }(\varvec{s}) = [ \varvec{\phi }(s_1)\cdot \varvec{w}, \ldots , \varvec{\phi }(s_q)\cdot \varvec{w}]^{T}\) is more easily expressed as \(\bar{V}^{\pi }(\varvec{s})= \varPhi (\varvec{s})\varvec{w}\) where \(\varPhi (\varvec{s})\) is a matrix with columns formed by the basis functions \(\phi _1, \phi _2,\ldots ,\phi _k\) and rows formed by the states such that \(\varPhi (\varvec{s})_{ij} = \phi _j(s_i)\). We further use the notation \(\varPhi _J\), where \(\varPhi \in {\mathbb {R}}^{q\times v}\) is a matrix and \(J \subset \{1,\ldots ,v\}\) is an index set, to mean the submatrix of \(\varPhi \) containing only the columns with indices present in J.
We use two types of basis functions, radial basis functions (RBFs) and the Fourier basis. An RBF is a Gaussian with variance \(\sigma ^2 > 0 \in {\mathbb {R}}\) and center \(b_i \in {\mathbb {R}}^d\): \(\phi _i(s) =\frac{1}{\sigma ^{d/2} \pi ^{d/4}} e^{-||b_i - s||^2/2\sigma ^2}\), where each element \(b_{ij}\) gives the RBF's center position in dimension \(j \in \{1,\ldots ,d\}\). Typically, for an order \(n\) RBF approximation, \(n+1\) RBFs are tiled evenly across each dimension with an additional constant function for a total of \((n+1)^d + 1\) basis functions. The RBF centers \(b_i\) in each dimension are further parametrized, with \(b_{ij} = (c_{ij})/n\), where \(c_i = [c_{i1}, c_{i2},\ldots , c_{id}], c_{ij} \in \{0,\ldots ,n\}\). The entire tiling can be obtained by varying c, and while \(\sigma \) can vary, we use \(\sigma ^2 = 1/(2(n+1)^3 - 4(n+1)^2 + 2(n+1))\) as described in Benoudjit and Verleysen (2003).
An \(n^{th}\) order Fourier basis set (Konidaris et al. 2011) consists of cosine functions of the form: \(\phi _i(s) = cos(\pi c_i \cdot s)\), where \(c_i = [c_{i1}, \ldots , c_{id}], c_{id} \in \{0,\ldots ,n\}\). By systematically varying c, the full order \(n\) set of \((n+1)^d\) basis functions can be obtained. In the context of the Fourier basis, we say \(\phi _i\) is independent if it is univariate.
2.1 Feature selection
The choice of features \(\phi _1,\phi _2,\ldots ,\phi _k\) can have a significant impact on the performance of RL algorithms. One way to choose these features is to use a complete fixed basis scheme, examples of which include CMAC tilings, RBF and Fourier basis sets. These sets must grow exponentially in size with the number of dimensions, quickly becoming impractical. An approximate sparse representation is often desirable (Kolter and Ng 2009), and therefore, methods for intelligently selecting sparse sets of features are of practical importance.
One method for feature selection that performs well is OMP-TD (Painter-Wakefield and Parr 2012). Despite lacking theoretical guarantees, its empirical performance is as good as or better than other feature selection methods such as LARS-TD (Kolter and Ng 2009; Painter-Wakefield and Parr 2012). For this reason, we only consider OMP-TD here, though our methods could be extended to other feature selection methods.
The OMP-TD algorithm is a variation of matching pursuit (MP) (Mallat and Zhang 1993). Given sampled states \(\varvec{s} = [s_1,\ldots ,s_q]^{T}\), weights \(\varvec{w} = [w_1, \ldots , w_k]^{T}\) and matrix \(\varPhi (\varvec{s})\in {\mathbb {R}}^{q\times k}, \varPhi (\varvec{s})_{ij} = \phi _j(s_i)\) made up of previously selected basis functions \(\phi _1,\ldots ,\phi _k\) evaluated on those states, the OMP-TD algorithm selects basis functions in the same greedy manner as MP. The basis function \(\phi _{k+1}\) is selected from a dictionary D of candidate basis functions such that the correlation \(\rho = |\langle \mathbf {R}_{k}, \varvec{\phi }(\varvec{x})\rangle |/||\varvec{\phi }(\varvec{x})||\) between the basis function and the residual error is maximised. OMP-TD uses the temporal difference (TD) error as the residual, \(\mathbf {R}_{k+1} = \varvec{r} + \gamma \varPhi (\varvec{s}')\varvec{w} - \varPhi (\varvec{s})\varvec{w}\), and maintains optimal weight values \(w_1,\ldots ,w_k\) using LSTD after each basis function is added. At each iteration, the residual is recalculated and the basis function in the dictionary with largest correlation to the residual is selected.
2.2 Regularization
An often desirable side effect of regularization is a sparse solution (Kolter and Ng 2009), which can be achieved directly with \(L_1\) regularization by imposing a sparsity inducing regularizer \(U(\varvec{w}) = ||\varvec{w}||_1\). \(L_2\) regularization instead introduces a constraint U which favors certain solutions or makes a problem well posed. \(L_2\) regularization has the advantage that solutions can be found analytically where \(L_1\) regularization often requires linear programming techniques. The most common form of constraint for \(L_2\) regularization, \(U(\varvec{w}) = ||\varGamma \varvec{w}||^2\), is known as Tikhonov regularization (Tikhonov 1963) or ridge regression, where \(\varGamma \) is a suitable regularization matrix (usually the identity matrix which encourages solutions with small weight norms). While it is common to choose a U which prefers low weights, U can also be chosen as a measure of the solution's smoothness. This expresses preference for simpler solutions that are less likely to overfit the data, and works in the incremental feature selection setting where basis functions with large weights are explicitly selected for.
3 Feature regularization
With OMP-TD, each basis function selected is guaranteed to improve the approximation provided it is sufficiently correlated with the Bellman error basis function (BEBF) (Parr et al. 2007), the basis function equal to the Bellman error. While correlation to the Bellman error is a reliable metric for feature selection, in practice OMP-TD can be slow to reduce prediction error when the value function does not resemble the reward function (Mahadevan and Liu 2010). Using the Neumann series expansion of the value function \(V = R + \gamma P R + \gamma ^2 P^2 R +\cdots \) it is easy to see as \(\gamma \) increases away from 0, the value function looks less like the reward function. This is important because the first \(k\) BEBFs which OMP-TD aims to select span the space of the first \(k\) terms of the Neumann series (Parr et al. 2008). Even values as low as \(\gamma =0.9\) can lead to slow convergence in practice (Mahadevan and Liu 2010). Since OMP-TD chooses a basis function which approximates the BEBF at each step, it can be even slower to converge.
One way to improve performance could be to choose basis functions less likely to overfit the BEBF, a problem when there are insufficient samples and many basis functions to chose from. This problem is particularly acute in the RL setting where dimensionality can be high and interaction costly. Another possibility is to introduce prior knowledge to augment selection. The smoothness prior can be used when the underlying value function is expected to be smooth. We show in Sect. 3.1 there is good reason to believe that value functions are smooth in general, making the smoothness prior a powerful tool for feature selection in RL.
We present feature regularization as a way to improve feature selection in RL by using regularization techniques in the selection process to incorporate prior knowledge/reduce overfitting. We present methods based on the OMP-TD algorithm because its strong empirical results on benchmark domains and simplicity make it a good choice for feature selection (Painter-Wakefield and Parr 2012). These methods could potentially be extended for use in other feature selection methods because they are decoupled from the LSTD fitting step in OMP-TD.
3.1 Regularized selection for smoothness
The idea that value functions are smooth in general is not new. Mahadevan and Maggioni (2007) note that value functions are typically smooth in Euclidean space. Their proto-value functions (PVFs), however, aim to take advantage of smoothness in the state graph. This type of smoothness derives from the fact that the value of a state is a function of "neighbouring" state values and associated rewards, where a state's neighbours are those reachable through transitions. Smoothness in Euclidean space is related, and coincides when neighbouring states are close in distance sufficiently often in conjunction with smooth or sparse reward functions. These conditions are often easy to fulfil as spatially local transitions are frequent in domains based on physical interactions (Jonschkowski and Brock 2013) and sparse/smooth reward functions are common.
Value function smoothness is also related to the discount factor \(\gamma \) by the Bellman equation, since the value of each state is a sum of \(\gamma \)-discounted values of future states. Values of \(\gamma \) closer to 1 lead to increased smoothness in the value function along trajectories through the state space as successive values change less. This however, can also lead to larger differences in values between states which are close in space but not close in the state graph.
3.2 Smoothness regularized feature selection methods
We describe two ways of integrating the smoothness regularizer into the selection process, first using the standard Tikhonov regularization formulation with STOMP-TD, and second by directly modifying the basis function correlation formula with the SSOMP-TD selection algorithm.
The algorithm for each method follows the same steps as OMP-TD. Given a set of samples obtained using a fixed policy, and a dictionary of basis functions, an approximation set of basis functions is formed. This is initialized with at least the constant function. Weights are fitted to the approximation set using LSTD, and the resulting Bellman residual calculated. The correlation \(\rho \) of the Bellman residual to each basis function in the dictionary is calculated. This correlation is then modified by STOMP-TD (Eq. 6) or SSOMP-TD (Eq. 7) and the single most correlated basis function is selected and added to the approximation set. The weights are again calculated by LSTD and the process repeated until the Bellman residual is sufficiently small.
The algorithms can be prevented from selecting redundant features by using orthogonal basis sets in the dictionary, however even with redundant features, their correlation to the Bellman error will be small. This is because LSTD is the orthogonal projection of TV into the approximation set, and consequently Parr et al. (2007) show that the Bellman residual (\(TV-V\)) is orthogonal to the span of the approximation set.
3.2.1 Smooth Tikhonov OMP-TD
3.2.2 Smoothness scaled OMP-TD
Both STOMP-TD and SSOMP-TD only modify the correlation calculation without increasing complexity and hence the running times are the same as OMP-TD's \(O(Kq)\) for selection, and \(O(k^3 + qk^2)\) for fitting with LSTD, where \(K\) is the size of the dictionary, \(q\) is the number of samples and \(k\) is the size of the approximation set.
3.3 ROMP-TD and LSTD-RP
We compare our methods to two methods with similar goals; the TD version of the regularized orthogonal matching pursuit algorithm (ROMP) (Needell and Vershynin 2009): ROMP-TD, and the least squares TD with random projections algorithm (LSTD-RP) (Ghavamzadeh et al. 2010).
An adaptation to the OMP algorithm, ROMP has not yet been applied in the RL setting to the best of our knowledge. While the theoretical guarantees of the ROMP algorithm may still apply in the TD case, we leave the proofs of such for a more complete review of the ROMP algorithm in this setting. We include it as the only other algorithm we know of which employs regularization in the selection step of OMP. Unlike our OMP-TD based methods, ROMP selects a set of basis functions to add to the approximation at each step instead of a single function. The selected set \(J_0\) fulfils two criteria: (1) the correlations of each basis function in the set are comparable,4 and (2) the magnitude or energy \(||\rho |_{J_0}||^2\) of the correlations in the set is the largest amongst all sets which fulfil criteria 1. This forces the correlation of basis functions in the solution to be closer to uniform and thus more regular (Needell and Vershynin 2009). Intuitively, it is better to have a set of good basis functions rather than one highly correlated basis function which may overfit the data. The selection step of the ROMP algorithm is decoupled from the OMP fitting step and can be inserted into OMP-TD without modification.
LSTD-RP deals with large feature sets differently. Instead of selecting features so that the set used in the fitting step is computationally manageable, LSTD-RP projects the entire set of features using a random projection matrix onto a low dimensional space. The low dimensional space results in a significant reduction in the size of the matrices used in LSTD, making it possible to compute value functions despite the curse of dimensionality.
The running time of the ROMP selection step is of the same order as OMP-TD's selection step5: \(O(Kq)\) where \(K\) is the number of basis functions in the dictionary and \(q\) is the number of samples.
LSTD-RP has the advantage that it is not incremental and the LSTD calculation only has to be done once in \(O(k^3 + qkK)\) where \(q\) is the number of samples, \(K\) is the size of the dictionary and \(k\) is the size of the low dimensional space which is the size of the approximation sets of the feature selection methods. Predicting the value of a state comes with a cost of \(O(Kk)\) instead of \(O(k)\) for feature selection methods.
4 Experiments
Our goal is to compare the selection methods based on the ordering they impose on their candidate basis functions. We therefore show the performance of each method against the number of basis functions selected instead of the stopping condition parameter (\(\beta \)) described in the OMP-TD algorithm (Painter-Wakefield and Parr 2012). Any setting of \(\beta \) would correspond to adding a vertical line at some point in the graph. Our experimental design is similar to that used in Painter-Wakefield and Parr (2012). For ROMP-TD, where sets of basis functions are selected,6 we added basis functions from the set one at a time in descending order of correlation magnitude to make the methods comparable. For LSTD-RP, we projected the entire dictionary available to the other algorithms such that the matrices used in the fitting step of each method were the same size. We used the same amount of \(L_2\) regularization as the other methods in the fit.
In the RBF experiments, the approximation set contained the constant function \(\phi _1=1\) at the start. The dictionary contained basis function sets up to a maximum order. For example, with a maximum order of 4, the RBF basis sets of orders \(n=1,2,3\) and 4 would all be added to the dictionary, with each set using a different variation parameter \(\sigma ^2 = 1/(2(n+1)^3 - 4(n+1)^2 + 2(n+1))\) depending on its order, \(n\). This would give the selection algorithm the choice between wider (more general/smoother) and thinner (more specific/sharper) basis functions where appropriate.
In the Fourier basis experiments, the approximation set contained independent basis functions up to a given order at the start. The dictionary contained all basis functions up to a maximum order. For the chain walk experiments, where all basis functions are independent, only the constant basis function was added to the approximation set at the start.
4.1 Domains
In each domain, training and test samples were collected using a fixed deterministic policy. In chainwalk, pairs of samples were collected by following the optimal policy from a random state to the next state. In the other domains, the given policy was followed from a random state until termination. The error in the approximation was obtained by comparing the approximate value \(\bar{V}^{\pi }\) to the true value \(V^{\pi }\) using the root mean squared error. In chainwalk, \(V^{\pi }\) was calculated using dynamic programming. In the other domains, \(V^{\pi }\) was calculated using a single rollout (deterministic policy).
Discount, initial dictionary and approximation set, samples collected and \(L_2\) regularization for each experiment using the RBF basis
\(\gamma \)
Max order
Initial BFs
Total BFs
Training samples
Test samples
\(L_2\) regularization
50 state chainwalk
Mountain car
Mountain car 3D
RC car
Discount, initial dictionary and approximation set, samples collected and \(L_2\) regularization for each experiment using the Fourier basis
\(\gamma \)
Max order
Initial BFs
Total BFs
Training samples
Test samples
\(L_2\) regularization
50 state chainwalk
Mountain car
Mountain car 3D
RC car
50 state chainwalk: the 50 state chainwalk domain (Lagoudakis and Parr 2003) is a 1 dimensional domain with 50 states and rewards at states 10 and 41. We scaled the states between 0 and 1 for function approximation.
Mountain car: mountain car (Sutton and Barto 1998) is a 2 dimensional domain where an underpowered car must escape a valley by driving up the sides, using its momentum to get to the top of the valley. We used the policy of accelerating in the direction of the current velocity or going right if at rest, used in Painter-Wakefield and Parr (2012).
Mountain car 3D: mountain car 3D (Taylor et al. 2008) is a 4 dimensional version of the Mountain Car domain. We used a similar policy to the 2 dimensional case where the car always accelerates in the direction of maximum velocity, or in the closest direction towards the goal if stationary.
RC Car: the remote control car domain (Geramifard 2013) is a 4 dimensional domain with 9 actions in which a remote control car must be maneuvered to within a distance of 0.1 m from the goal. The 9 possible actions are derived from combinations of 2 primitive action types: steering \(\in \{ \mathtt {left}, \mathtt {straight}, \mathtt {right}\}\) and acceleration \(\in \{\mathtt {forward}, \mathtt {coast}, \mathtt {backward}\}\). The state is given by the x and y position of the car, its heading \(\theta \) in radians and its velocity v. If the car collides with any of the walls in the \(2\times 3\) m room it's in, its velocity is set to 0, otherwise the domain is frictionless. At each step a reward of \(-1\) is given if the agent is not at the terminal goal state or 100 if it is. The policy \(\pi \) used simulates all possible actions at each state and chooses the one which minimises \(\delta + 2o\), where \(\delta \) is the Euclidean distance to the goal and o is the orthogonal distance to the goal along the current trajectory of the car.
Acrobot: the acrobot domain (Sutton and Barto 1998) is a 4 dimensional domain where an agent controls the torque of the middle link of a 2 link pendulum. The goal is to raise the tip of the pendulum a certain height. The policy applies torque in the direction of the joints if they are moving in the same direction, else it reverses it. Occasionally it didn't terminate, and we discarded those samples.
4.2 Results
The SSOMP-TD algorithms performed reliably well, with the smoothing regularizer \(U_1\) performing the worst of the three on average, and \(U_2\) most often performing the best, or tied for best. SSOMP-TD performed its worst on acrobot with both basis sets. This could indicate that the underlying value function of our policy is not smooth. The algorithm seems to be at least as good as the others, if not better on the remaining domains taken as a whole.
The STOMP-TD algorithm performed well across all domains, however selecting an appropriate \(\lambda \) was difficult, and some settings resulted in performance significantly worse than OMP-TD. It is more powerful than SSOMP-TD as it can be adapted to use any prior, but it is also more difficult to use. In practice, we would use SSOMP-TD as choosing the m parameter is significantly easier than choosing \(\lambda \), and the results are comparable for the two methods.
ROMP-TD was never the best performer, and in multiple experiments performed the worst. ROMP-TD primarily tackles the problem of overfitting, and it is possible there were too many samples for overfitting to occur. This suggests that the effectiveness of SSOMP-TD and STOMP-TD is primarily due to the smoothness of the value function in these experiments. Interestingly in the Fourier pinball experiment, selecting basis functions randomly outperformed ROMP-TD. It is possible that selecting batches of basis functions was too conservative and the other methods could take advantage of the changing residual after each selection. Even randomly chosen functions were better than a set of functions selected to approximate one residual well.
LSTD-RP performed poorly in all domains, only beating the random selection method in two of the RBF experiments. Its best performance with the Fourier basis was in mountain car, which was the only experiment where every basis function in the dictionary was selected. In that experiment its performance matched that of the other methods near the end, indicating that LSTD-RP is negatively affected by poor basis functions in the dictionary, which dilute the resulting projection.
The spikes in the graphs, particularly in Fig. 5c are the result of fitting problems with LSTD. We verified this by fitting the basis set at each step to the estimates of the true value function obtained using rollouts. We used the same training and test set and plotted the results.7 As expected, the error for each method decreased monotonically and the relative performance of the methods was similar.
5 Future work and conclusions
A major concern when using function approximation in high dimensional domains is the large computational costs involved. Feature selection, the automated process of selecting a subset of these functions to use in the approximation, has consequently begun to receive significant attention recently. However, current greedy selection techniques ignore characteristics of the underlying value function and select basis functions which best fit the sample data. This may lead to the selection of more basis functions than necessary. The introduction of prior knowledge can improve performance. One such prior is the smoothness prior which can be integrated into the selection process via regularization.
We have presented three ways to regularize feature selection in RL, ROMP-TD, STOMP-TD and SSOMP-TD. Our results using these methods show that feature regularization is a simple yet powerful tool for improving performance without significant computational overhead. Further, the smoothness prior generalizes the intuition that wide/low frequency basis functions are better when only a few basis functions can be selected, and shows continued empirical improvements well beyond the first few basis functions.
One possible direction for future study is to combine STOMP-TD and SSOMP-TD with state space restructuring methods such as Predictive Projections (Sprague 2009), which projects the original state space in such a way that neighboring states in the state graph are closer together in the projected state space. There is also potential to use the smoothness prior in dictionary construction, to either reduce its size significantly in high dimensional domains, or to improve the quality of basis functions in a dictionary of fixed size. Randomly selected RBFs would be a natural choice, where a bias towards smooth RBFs could be included easily.
- 1.
There are many ways to measure smoothness, many of which exhibit similarities, for example the total variation norm is similar to \(U_1\), where instead of squaring the function's derivative, the absolute value is taken instead. Our smoothness measure was chosen for its intuitive simplicity and existence of analytic solutions.
- 2.
Derivations for these two equations are given in the "Appendix".
- 3.
The regularized correlation is derived in the "Appendix".
- 4.
The correlation \(\rho _i\) of basis function \(\phi _i\) is comparable to the correlation \(\rho _j\) of \(\phi _j\) if \(|\rho _i| \le 2|\rho _j|\) and \(|\rho _j| \le 2|\rho _i|\).
- 5.
See the "Appendix" for the complete algorithm and analysis.
- 6.
We did not restrict the size of the selected set using the optional parameter \(\psi \).
- 7.
Many of the plots involving the true error \(||V^\pi - \Pi V^\pi ||\) overlapped the others, so we have omitted them for clarity.
- Benoudjit, N., & Verleysen, M. (2003). On the kernel widths in radial-basis function networks. Neural Processing Letters, 18, 139–154.CrossRefGoogle Scholar
- Bradtke, S., & Barto, A. (1996). Linear least-squares algorithms for temporal difference learning. Machine Learning, 22(1), 33–57.zbMATHGoogle Scholar
- Dabney, W., & Barto, A. (2012). Adaptive step-size for online temporal difference learning. In Twenty-Sixth AAAI Conference on Artificial Intelligence (pp. 872–878).Google Scholar
- Ernst, D., Geurts, P., & Wehenkel, L. (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 503–556.Google Scholar
- Geist, M., Scherrer, B., Lazaric, A., & Ghavamzadeh, M. (2012). A Dantzig selector approach to temporal difference learning. In Proceedings of the 29th international conference on machine learning (pp. 1399–1406).Google Scholar
- Geramifard, A. (2013). Rc-car domain. http://acl.mit.edu/RLPy/api/domains_misc.html#rccar
- Ghavamzadeh, M., Lazaric, A., Maillard, O., & Munos, R. (2010). LSTD with random projections. Advances in Neural Information Processing Systems, 23, 721–729.Google Scholar
- Ghavamzadeh, M., Lazaric, A., Munos, R., & Hoffman, M. (2011). Finite-sample analysis of Lasso-TD. In Proceedings of the 28th international conference on machine learning (pp. 1177–1184).Google Scholar
- Johns, J., Painter-Wakefield, C., & Parr, R. (2010). Linear complementarity for regularized policy evaluation and improvement. Advances in Neural Information Processing Systems, 23, 1009–1017.Google Scholar
- Jonschkowski, R., & Brock, O. (2013). Learning task-specific state representations by maximizing slowness and predictability. In 6th international workshop on Evolutionary and Reinforcement Learning for Autonomous Robot Systems (ERLARS).Google Scholar
- Kolter, J., & Ng, A. (2009). Regularization and feature selection in least-squares temporal difference learning. In Proceedings of the 26th annual international conference on machine learning (pp. 521–528).Google Scholar
- Konidaris, G., & Barto, A. (2009). Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in Neural Information Processing Systems, 22, 1015–1023.Google Scholar
- Konidaris, G., Osentoski, S., & Thomas, P. (2011). Value function approximation in reinforcement learning using the Fourier basis. In Proceedings of the twenty-fifth conference on artificial intelligence (pp. 380–385).Google Scholar
- Lagoudakis, M., & Parr, R. (2003). Least-squares policy iteration. Journal of Machine Learning Research, 4, 1107–1149.MathSciNetGoogle Scholar
- Mahadevan, S., & Liu, B. (2010). Basis construction from power series expansions of value functions. Advances in Neural Information Processing Systems, 23, 1540–1548.Google Scholar
- Mahadevan, S., & Maggioni, M. (2007). Proto-value functions: A laplacian framework for learning representation and control in markov decision processes. Journal of Machine Learning Research, 8, 2169–2231.Google Scholar
- Mallat, S., & Zhang, Z. (1993). Matching pursuits with time–frequency dictionaries. IEEE Transactions on Signal Processing, 41(12), 3397–3415.CrossRefzbMATHGoogle Scholar
- Needell, D., & Vershynin, R. (2009). Uniform uncertainty principle and signal recovery via regularized orthogonal matching pursuit. Foundations of Computational Mathematics, 9(3), 317–334.MathSciNetCrossRefzbMATHGoogle Scholar
- Painter-Wakefield, C., & Parr, R. (2012). Greedy algorithms for sparse reinforcement learning. In Proceedings of the 29th international conference on machine learning (pp. 1391–1398).Google Scholar
- Parr, R., Painter-Wakefield, C., Li, L., & Littman, M. (2007). Analyzing feature generation for value-function approximation. In Proceedings of the 24th international conference on Machine learning (pp. 737–744).Google Scholar
- Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., & Littman, M. (2008). An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In Proceedings of the 25th international conference on machine learning (pp. 752–759).Google Scholar
- Petrik, M., Taylor, G., Parr, R., & Zilberstein, S. (2010). Feature selection using regularization in approximate linear programs for Markov decision processes. In Proceedings of the 27th international conference on machine learning (pp. 871–878).Google Scholar
- Sprague, N. (2009). Predictive projections. In Proceedings of the 21st international joint conference on Artifical intelligence (pp. 1223–1229).Google Scholar
- Sun, Y., Gomez, F.J., Ring, M.B., & Schmidhuber, J. (2011). Incremental basis construction from temporal difference error. In Proceedings of the 28th international conference on machine learning (pp. 481–488).Google Scholar
- Sutton, R., & Barto, A. (1998). Introduction to reinforcement learning. Cambridge: MIT Press.Google Scholar
- Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9–44.Google Scholar
- Taylor, M., Kuhlmann, G., & Stone, P. (2008). Autonomous transfer for reinforcement learning. In The seventh international joint conference on autonomous agents and multiagent systems (pp. 283–290).Google Scholar
- Tikhonov, A. (1963). Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics Doklady, 5, 1035.zbMATHGoogle Scholar
- Van Roy, B. (1998). Learning and value function approximation in complex decision processes. PhD thesis, Massachusetts Institute of Technology.Google Scholar
- Wahba, G. (1990). Spline models for observational data (Vol. 59). Philadelphia: SIAM.Google Scholar
- Yu, H., & Bertsekas, D. P. (2009). Convergence results for some temporal difference methods based on least squares. Automatic Control, IEEE Transactions on, 54(7), 1515–1531.MathSciNetCrossRefGoogle Scholar


text:
|Preparation of pure human tumor necrosis factor and hybridomas producing monoclonal antibodies to human tumor necrosis factor.
This is a continuation of application No. 07/759,976 filed Sep. 17, 1991, now abandoned, which is a continuation of application No. 07/427,277 filed Oct. 24, 1989, now abandoned.
(1) is an IgG1 of a molecular weight of about 155 Kilo-daltons, which comprises a heavy chain having a molecular weight of about 53 Kilo-daltons and a light chain having a molecular weight of about 24.5 Kilo-daltons;
(2) specifically reacts with SDS-denatured human tumor necrosis factor alpha; and
(3) has neutralizing activity against human tumor necrosis factor alpha in a human tumor necrosis factor alpha bioassay.
(1) is an IgG1 of a molecular weight of about 155 Kilo-daltons, comprising a heavy chain having a molecular weight of about 53 Kilo-daltons and a light chain having a molecular weight of about 24.5 Kilo-daltons;
(2) specifically reacts with SDS-denatured human tumor necrosis factor alpha; and
(3) has neutralizing activity against human tumor necrosis factor alpha in a human tumor necrosis factor alpha bioassay.
(A) exposing a sample to monoclonal antibody ANOC 705 secreted by hybridoma KOCO 705 having Fermentation Research Institute Deposit No. FERM BP-2569, wherein said monoclonal antibody:
(1) is an IgG1 of a molecular weight of about 155 Kilo-daltons, which comprises a heavy chain having a molecular weight of about 53 Kilo-daltons and a light chain having a molecular weight of about 24.5 Kilo-daltons;
(2) specifically reacts with SDS-denatured human tumor necrosis factor alpha;
(3) has neutralizing activity against human tumor necrosis factor alpha in a human tumor necrosis factor alpha bioassay; and
(B) detecting the presence of human tumor necrosis factor in said sample by determining whether said monoclonal antibody ANOC 705 has bound to antigen.
1. Field of the Invention
The present invention relates to monoclonal antibodies for human TNFs (tumor necrosis factors), and, more particularly, to monoclonal antibodies for a human TNF which are useful as a medicine and ensure immunological purification and measurement of said human TNF.
2. Description of the Background Art
A lipopolysaccharide (LPS), when administered to a mouse, rabbit, or rat sensitized by Bacillus de Calmette Guerin (BCG), induces a serum factor which hemorrhagically necroses tumors. Carswell et al. named this factor "TNF" in 1975 [Proc. Nat. Acad. Sci., USA, 72. 3666 (1975)]. Macrophages proliferate in the spleen of a mouse sensitized with BCG but elapse by the administration of LPS. From this, TNF has been considered to be produced by macrophages. Recent studies have unveiled the fact that an in vitro LPS treatment of isolated macrophages induced a TNF activity in the supernatant of the macrophage culture broth, confirming that macrophages are TNF-producing cells. Several leukemia cells producing TNF have also been reported.
The characteristic of TNF destroying various cancer cells and inhibiting their growth, while exhibiting no such effects on normal cells, suggests its potential utility as a carcinostatic agent. The anti-tumor activity of TNF has been confirmed in experiments using a purified TNF derived from mice and rabbits.
The TNF activity has been measured using a fibroblast cell line, L929, having a strong sensitivity to a TNF. The TNF unit of activity is generally expressed as the amount which results in a 50% cytotoxicity of L929 cells in a culture plate.
The molecular weights of TNF produced by various animals as determined by a gel filtration analysis are reported to be 150,000 and 40,000-60,000 for TNF produced by mouse, 67,000 and 39,000 for TNF produced by rabbit, and 34,000-140,000 for TNF produced by human. According to molecular weight determination using SDS-PAGE, purified TNFs derived from rabbit and human are reported to be 17,000. TNFs having this molecular weight are termed "TNF-α" or tumor necrosis factor-alpha. Natural TNFs are considered to exist as oligomers.
In 1984, the cloning of TNF-cDNA was successfully achieved using human myeloma leukemia cells "HL-60" which are a type of TNF-producing cell. This has ensured large-scale production of TNF using E. coli. The cDNA cloning revealed the fact that human TNF is constituted by 157 amino acids and has a long preceding polypeptide having 76 amino acids [D. Pennica et al., Nature, 312, 724-729 (1984)]. Very closely following this, the successful cloning of TNF chromosomal genes was also reported [T. Shirai et al., Nature, 313, 803-806 (1985)], and it was revealed that human TNF genes are composed of four exons.
The amino acid sequence of TNF and the gene base sequence have respectively 28% and 46% homologies with a cytotoxin factor and a lymphotoxin (LT) produced by B-cells. TNF, however, has a substantive difference from LT in that the former has no N-glycosil-type sugar chain bonding site. Although TNF and LT have no immunological cross-reactivity, they exhibit very similar cytotoxicity. The homological portions of TNF and LT are considered to be involved in the cytotoxicity.
As mentioned above, because of its special physiological activity, the utility of TNF as a medicine can be expected. A number of studies have been undertaken with a view to utilize TNF as a medicine. Also, extensive research is are underway on the subject of various immunodeficiencies and immunoreactions, as well as on the subject of TNF assay in clinical samples of diseases related to immunodeficiencies and immunoreactions.
A bioassay technique is currently used for the analysis of the aforementioned TNF-α. According to this method TNF is measured in terms of its activity. The method, however, has problems in its complicated procedure and insufficient accuracy. The method also requires stringent precaution concerning any components interfering with the assay result. Development of a measurement technique substituting for the bioassay technique has therefore been sought.
Accordingly, an object of the present invention is to provide monoclonal antibodies exhibiting specific reactivities to human TNF-α which are useful as a medicine and ensure immunological purification and measurement of said human TNF.
Other objects, features and advantages of the invention will hereinafter become more readily apparent from the following description.
According to the present invention monoclonal antibodies having a specificity for human TNF-α is provided.
A novel and simple immunoassay technique for measuring said human TNF-α with high sensitivity and with high precision can be provided by the use of the monoclonal antibodies of the present invention.
Since the monoclonal antibodies of the present invention are specific with respect to TNF-α, they can be utilized for a specific purification of TNF-α if applied to affinity chromatography or the like.
The antibodies of the present invention are hereinafter described in detail.
The antibodies of the present invention can be prepared by using a TNF-α as an antigen. More specifically, a hybridoma is first prepared from mammal phlogocytes (immunocytes) sensitized with said antigen and mammal plasmacytoma cells. A clone capable of producing a desired antibody is selected from the hybridoma, cultured, and collected.
There are no specific limitations as to the types of TNF-α used as an antigen in this method. A supernatant of a culture medium containing a human TNF induced by the application of a known in vitro method or its purified product [Proc. Natl. Acad. Sci., USA, 82, 6637 (1985)], a human TNF produced by the use of a gene recombinant technique [Nature, 312, 724-729 (1984)], or a synthetic peptide prepared having a part of the amino acid sequence of one of these TNFs can be used.
There are no specific limitations as to mammals to be immunized with the antigen. It is desirable, however, that the mammal be selected taking into consideration its compatibility with plasmacytoma cells used for the cell fusion. Usually, the use of mouse or rat is advantageous.
Immunization can be performed by administering said antigen to a mammal by intravenous, subcutaneous, intradermic, or intraperitoneal injection. More specifically, the antigen, if desired, mixed with a conventional adjuvant, is administered to an animal, e.g. mouse, once a day at 3-5 intervals until the total amount administered becomes about 100-500 μg/mouse. Preferably immune spleen cells are extracted about 3 days after the above final administration.
There are several cells known in the art as plasmacytoma cells of mammals which can be used as the parent cells to be fused with the immune cells. They are, for example, myeloma tumor cells such as p3 (p3/×63-Ag8) [Nature, 256, 495-497 (1975)], p3-U1 [Current Topics in Microbiology and Immunology, 81, 1-7 (1978)], NS-1 [Eur. I. Immunol., 6, 511-519 (1976)], MPC-11 [Cell, 8, 405-415 (1976)], SP2/O [Nature, 276 269-270 (1978 ) ], FO [J. Immunol. Meth., 35, 1-21 (9180)], ×188.8.131.52. [J. Immunol., 123, 1548-1550 (9179)], S194 [J. Exp. Med., 148, 313-323 (1978)], etc., rat R210 [Nature, 277, 131-133 (1979)], and the like.
Fusion of these plasmacytoma cells and the above immune cells can be carried out according to a known method, for example, according to the method of Milstein [Method in Enzymology, 73, 3 (1981)]. Specifically, the cell fusion can be carried out in a normal culture medium in the presence of a conventional cell fusion promoter such as polyethylene glycol (PEG), Sendai virus (HVJ), etc. An auxiliary agent such as dimethylsulfoxide may be added to the culture medium, if necessary, for further promoting the cell fusion efficiency. The ratio of plasmacytoma cells and immune cells used may be a ratio commonly employed in the art. Usually, immune cells are used in an amount of 1-10 times plasmacytoma cells. Various media commonly used for culturing plasmacytoma cells, RPMI-1640, MEM etc., can be used for the cell fusion. It is desirable that serum adjuncts such as fetal cattle serum (FCS) be eliminated from the medium. The cell fusion is carried out by thoroughly mixing a prescribed amount of plasmacytoma cells and immune cells in a medium and by adding to it about 30-60 w/v% of a PEG solution having an average molecular weight of 1,000-6,000 which is heated in advance to about 37° C. Thereafter, the addition of a suitable medium and centrifugation to remove the supernatant are repeated until a desired hybridoma is formed.
Separation of the hybridoma can be carried out by culturing the hydbridoma in a conventional selection medium, such as, for example, HAT medium (a medium containing hypoxanthine, aminopterin, and thymidine). The culture is carried out for a period of time, usually several days to several weeks, sufficient to destroy cells other than target hybridoma such as unfused cells and the like. The hybridoma thus produced is subjected to screening of the target antibody by means of a conventional limiting dilution analysis and to monocloning.
The screening of the target antibody can be carried out by any one of various methods commonly used for antibody screening, such as the ELISA method [Engvall, E., Meth. Enzymol., 70, 419-439 (9180)], the plaque method, the spot method, the agglutination reaction method, the Ouchterlony method, the radioimmunoassay (RIA) method, and the like ["Hybridoma and Monoclonal Antibody", R&D Planning Publishing Co., 30-53 (1982)]. The above-mentioned antigen can be used for the screening.
The hybridoma thus obtained capable of producing desired monoclonal antibodies which can identify human TNF-α can be subcultured in a normal medium and can be stored in liquid nitrogen for a long period of time.
Collection of desired antibodies from the hybridoma can be performed, for example, by culturing the hybridoma according to a conventional method and collecting the supernatant containing antibodies, or by administering the hybridoma to a mammal having compatibility therewith, allowing the hybridoma to grow, and collecting the antibodies from the abdominal dropsy. The former method is suitable for preparing a high-purity antibody and the latter can be applied to a large-scale production of antibodies.
The antibodies thus produced can be purified by a conventional method such as salting-out, gel permeation, affinity chromatography, or the like.
The monoclonal antibodies of the present invention thus prepared have a specificity for TNF-α.
Among the antibodies of the present invention, there is one type having a neutralizing effect on human TNF-α biological activity. This type of antibody is particularly suitable for the specific measurement of human TNF-α having a biological activity. Also, among the antibodies of the present invention, there is a type which is capable of recognizing specifically different sites of TNF-α, exhibits no steric hindrance between themselves, and, at the same time, can combine with TNF-α. Such an antibody is useful in a sandwich assay and the like. In addition, there are antibodies of the present invention which exhibit a particularly superior reactivity in liquid-phase or solid-phase systems. This type of antibody is suitable for use in liquid- or solid-phase immunoassay.
According to the present invention monoclonal antibodies which have a specificity to human TNF-α are provided. Also, an immunoassay technique having extremely high sensitivity and superior specificity can be provided by the use of antibodies of the present invention. The immunoassay technique can detect and measure with a very high precision human TNF-α in samples such as clinical samples containing even an extremely small amount of human TNF-α.
Other features of the invention will become apparent in the course of the following description of the exemplary embodiments which are given for illustration of the invention and are not intended to be limiting thereof.
Preparation of Antibodies of the Present Invention
A human TNF-α (10-20 [g) prepared by a gene recombinant technique [Nature, 312, 724-729 (1984)] was intraperitoneally administered to a BALB/c mouse together with complete Freund's adjuvant. The same amount of antigen was administered twice together with incomplete Freund's adjuvant, first 3-4 weeks after the initial administration and then 3-4 weeks thereafter. After 3-4 days following the last administration, cell fusion was performed according to a conventional method [Method in Enzymology, 73, 3 (1981)], using immunized spleen cells and myeloma tumor cells [P3U1, Current Topics in Microbiology, 81, 1-7 (1978)] at a ratio of 10:1 in polyethylene glycol (PEG-4000). The hybridoma was screened with HAT medium. The supernatant was tested by an enzymatic immunoassay technique using a 96-well microplate coated with said human TNF-α and a (goat) peroxidase-labeled anti-mouse IgG (product of E. Y. Laboratories, Inc.), thereby detecting cells capable of producing the target antibody for the human TNF-α.
Cloning by a limiting dilution technique was repeated to produce 7 clones capable of producing the desired antibodies. They were named KOCO701, KOCO702, KOCO703, KOCO704, KOCO705, KOCO706, and KOCO707. The antibodies produced by them were respectively named ANOC701, ANOC702, ANOC703, ANOC704, ANOC705, ANOC706, and ANOC707.
One of the clones, "KOCO 705", was deposited on Jun. 1, 1988 at the Fermentation Research Institute 1-3, Higashi, 1 chome, Tsukuba-shi, Ibaraki-ken 305, Japan under the name of KOCO 705 (FERM BP-2569).
Characteristics of the antibodies produced by each of the above clones are as follows:
(1) Antibody Subclass
The subclass was determined using a mouse antibody subclass detection kit (product of Bio-Rad Co.). The results are given in Table 1.
|Antibody No. Subclass
ANOC701 IgG1
ANOC702 IgG1
ANOC703 IgG2a
ANOC704 IgG1
ANOC705 IgG1
ANOC706 IgG1
ANOC707 IgG1
(2) Antibody Production Level
IgG in ascites was purified using the Protein A affinity, and concentration (mg/ml) was determined by OD280 adsorption. The OD280 value at a 1 mg/ml IgG concentration was assumed to be 1.4.
The results are shown in Table 2.
|Antibody No. IgG (mg/ml)
(3) Neutralization Antibody Titer
The neutralization antibody titer of the ascites was determined by the LM assay technique which is a type of TNF bioassay. The neutralization antibody titer is defined as the number of a TNF-α units which can be neutralized by 1 ml of ascites.
The results are shown in Table 3.
|Neutralization Antibody No. Antibody Titer (U)
(4) Molecular Weight
Antibodies obtained from ascites were purified using an IgG purification kit (MOPS Kit, product of Bio-rad Co.). The IgG's heavy-chain and light-chain molecular weights were determined by electrophoresis using SDS-PAGE in the presence of 2 ME. The sum of the two molecular weights was taken as representing one-half of the molecular weight of the intact monoclonal antibodies.
The results are shown in Table 4.
|Molecular Weight Antibody No. Heavy chain Light chain IgG
(5) Western Blotting Analysis
Specificity for TNF-α and non-reactivity with E. coli. of the antibodies of the present invention were confirmed in the following manner. τTNF-α and τTNF-α-producing E. coli were treated with an SDS lysis buffer to a final concentration of 100 μg/ml. SDS-PAGE was performed to effect blotting onto nitrocellulose and to determine the reactivity of each monoclonal antibody.
The results are shown in Table 5.
|Reactivity Antibody No. E. coli protein TNF-α
Table 5 shows that ANOC701, ANOC702, ANOC704, ANOC705, ANOC706, and ANOC707 combine with TNF-α with specificity and do not react with E. coli protein. The non-reactivity of ANOC703 with τTNF-α suggests its recognition of a steric structure.
(6) ELISA Sensitivity
Each 100 μl of monoclonal antibodies ANOC701-ANOC 707 diluted with PBS to a concentration of 10 μg/ml was poured into each well of a 96-well microplate and left overnight. The wells were blocked with 1% skim milk. 100 μl of a standard TNF-α was added to each well and incubated at 4° C. overnight. After washing the plate, 100 μl of a 1,000-fold solution of a rabbit polyclonal antibody (OCT701) against TNF-α was added and incubated for 2 hours at room temperature. The plate was washed and 100 μl of (goat) POD-labeled anti-rabbit IgG was added to each well. After incubation for 2 hours at room temperature, the plate was washed and the enzymatic activity of POD was measured.
The ELISA sensitivity of a monoclonal antibody is expressed by the TNF-α concentration of the antibody, the difference of which OD492 adsorption and the OD492 adsorption without TNF-α is 0.1.
The results are shown in Table 6.
|Antibody No. ELISA Sensitivity (OD492 = 0.1)
ANOC702 1.0 ng/ml (0.10 ng/well)
ANOC704 1.2 ng/ml (0.12 ng/well)
ANOC705 0.44 ng/ml (0.044 ng/well)
ANOC706 0.86 ng/ml (0.086 ng/well)
ANOC707 1.0 ng/ml (0.10 ng/well)
(8) Standard Curve by ELISA Method
100 μl of monoclonal antibody ANOC705 diluted with PBS to a concentration of 10 μg/ml was poured into each well of a 96-well microplate and left overnight at 4° C. Wells were blocked with 1% skim milk. 100 μl of a standard TNF-α was added in triplicate to each well and incubated at 4° C. overnight. After washing the plate, 100 μl of a 1,000-fold solution of a rabbit polyclonal antibody (OCT701) against TNF-α was added and incubated for 2 hours at room temperature. The plate was washed and 100 μl of (goat) POD-labeled anti-rabbit IgG was added to each well. After incubation for 2 hours at room temperature, the plate was washed and the enzymatic activity as OD492 adsorbance was measured using o-phenylenedimine as a substrate.
The results are shown in Table 7.
|TNF-α 0 9.375 37.50 150.0 600.0 concentration (pg/ml) Adsorbance 1 89 117 205 535 1510 Adsorbance 2 95 123 206 540 1481 Adsorbance 3 95 124 209 514 1487 Mean 93 121 207 530 1490 Mean - Blank 0 28 114 437 1397 C. V. (%) 3.7 3.1 1.0 2.6 0.7
Obviously, numerous modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

subset: commoncrawl


text:
About a year ago this mirror mysteriously appeared in our house. Nobody claimed it and nobody knew where it came from. In all honesty we're pretty sure it was a dumpster find. Despite it's shadowy origins, we found ourselves using it all the time. There was even a minor uproar when it disappeared for a day.
All I know is that I wanted to paint it the instant I saw it. The blank spaces were screaming to be filled in with color. Something whimsical and feminine, kinda like the Women's Building in San Francisco.
Finally, after staring at it for a year, I got out my acrylic paint and got to work. I wanted to incorporate a quote by Khalil Gibran but the space for it was limited and weirdly shaped. I thought it would look odd with a quote in just one corner so I scoured the internet looking for a few more empowering quotes about beauty.
It was surprisingly difficult. Various writers, celebrities, and sages have said many funny, inspiring, and thought provoking things about beauty (GoodReads alone has over 1,000 matches for the word "beauty.") What surprised me was how many of the quotes seemed to feed negative ideas about beauty: Beauty fades with age. ('Cause old people can't be beautiful?) Some of them suggested that beauty is only attainable if you are happy, dull, useless, or wearing the right accessory. In the end I found quite a few that really made me think about beauty; both what I believe about it and how it plays a role in our culture. I'll let you ruminate over them.
"There are no bad pictures; that's just how your face looks sometimes."
~ Abraham Lincoln
"Flawed, we're truly interesting, truly memorable, and yes, truly beautiful."
~ Justina Chen, North of Beautiful
"Remember that the most beautiful things in the world are the most useless; peacocks and lilies for instance."
~ John Steinbeck
"Let us leave pretty women to men with no imagination."
~ Marcel Proust, The Captive & The Fugitive
"I'm tired of all this nonsense about beauty being only skin-deep. That's deep enough. What do you want, an adorable pancreas?"
~ Jean Kerr
"Taught from their infancy that beauty is woman's sceptre, the mind shapes itself to the body, and roaming round its gilt cage, only seeks to adorn its prison."
~ Mary Wollstonecraft, A Vindication of the Rights of Woman
"A girl should be two things: classy and fabulous."
~ Coco Chanel
"Beauty, to me, is about being comfortable in your own skin. That, or a kick-ass red lipstick."
~ Gwyneth Paltrow
"A cultural fixation on female thinness is not an obsession about female beauty but an obsession about female obedience."
~ Naomi Wolf
"Beauty is in the eye of the beholder and it may be necessary from time to time to give a stupid or misinformed beholder a black eye."
~ Miss Piggy, The Muppets
"Screw pretty. I'd rather be strong. Pretty fades over time. Strength gets you through the bad shit."
~ Thea Harrison, Oracle's Moon
"Outer beauty attracts, but inner beauty captivates."
~ Kate Angell, Squeeze Play
"It's all mirror, mirror on the wall because beauty is power, the same way money is power, the same way a gun is power."
~ Chuck Palahniuk, Invisible Monsters
"I look my best after an entire hair and makeup team has spent hours perfecting me. When do I feel my best? When I haven't looked in a mirror for days, and I'm doing things that make me happy."
~ Anne Hathaway
"It is amazing how complete is the delusion that beauty is goodness."
~ Leo Tolstoy, The Kreutzer Sonata
"Beauty isn't worth thinking about; what's important is your mind. You don't want a fifty-dollar haircut on a fifty-cent head."
~ Garrison Keillor
"Crying is for plain women. Pretty women go shopping."
~ Oscar Wilde
"Taking joy in living is a woman's best cosmetic."
~ Rosalind Russell
"There is no exquisite beauty… without some strangeness in the proportion."
~ Edgar Allan Poe
"I believe in manicures. I believe in overdressing. I believe in primping at leisure and wearing lipstick."
~ Audrey Hepburn
"Makeup can only make you look pretty on the outside, but it doesn't help if you're ugly on the inside. Unless you eat the makeup."
~ Audrey Hepburn
"Beauty is how you feel inside, and it reflects in your eyes. It is not something physical."
~ Sophia Loren
I loved working on this project. It was exciting to have a project that challenged me to really think about the subject. I hope this mirror will bring as much inspiration and happiness to the people who look into it as it has to me. I kinda wish I had more junky mirrors lying around.
P.S. I decided that I loved the mirror enough that I would be upset if the paint job got scratched. I didn't have anything to seal it with so I officially broke the rules of the Destash Bash and bought a small bottle polyurothane varnish for $3.99. I may have contradicted the wisdom of Justina Chen, but I regret nothing.
P.P.S. Just in case you can't read them the quotes on the mirror are:
"Beauty is not in the face; beauty is a light in the heart."
~ Khalil Gibran
"To start telling people that you're beautiful, or just feel beautiful, just start acting like you are the most beautiful woman in the world."
~ Margaret Cho
"The best and most beautiful things in the world cannot be seen or even touched – they must be felt with the heart."
~ Helen Keller
"Everything has its beauty but not everyone sees it."
~ Confucius
"Beauty is whatever gives joy."
~ Edna St. Vincent Millay
"Beauty is about being comfortable in your own skin. It's about knowing and accepting who you are."
~ Ellen DeGeneres, Seriously…I'm Kidding


text:
I recently received the And I Don't Care: Whipped Hair & Body Moisturizing Butter from Heritage 1933, and it is another winner. This $30 whipped product works great on both the hair and the body and is made from natural ingredients. Heritage 1933 is a company that is trying to empower women. For ever purchase, the company will give free hair care products to women and children in need. They also have other bath, beauty, and hair products on their website. Make sure you check out their body scrubs and kinky curly weave.
Tag Archives: natural hair products
Beauty Bit- Unboxing and Trying My New Schwarzkopf Ultime Hair Products
I received these Schwarzkopf Ultime products complimentary from Influenster for testing purposes a few weeks ago. I shot the unboxing video, but I didn't get to post it. You can find these products exclusively at your local Walmart. Supermodel Claudia Schiffer helped create the line. I was worried that these products wouldn't work on my natural African-American textured hair, but at least one of them did.
Beauty Bit- Lottabody Curl and Style Milk Review
Y'all I finally bought the $5 bottle of Lottabody Curl and Style Milk. I now see why everyone was raving about it. This product keeps my hair sooooo soft and manageable. You can use it for twist outs too. That is what I did. I used it on my twists and on my wash and gos.
Beauty Bit- Murray's Edgewax is the Affordable Alternative to Hick's Edges
Murray's Edgewax 100% Australian Beeswax is the affordable edge control solution for natural hair. I found it at Walmart for around $3. This is like the low-cost alternative to Hick's Edges. I swear it works. I have like 4 category natural hair. I actually have two different textures, and it smooths down.
Beauty Bit- As I Am Coconut Cowash Review
I finally bought the As I Am Coconut Cowash. I know I am late to the party, but I love this stuff. It cleaned all of the gooky product I had left in my hair and left it incredible soft. I bought it for $9, but the website site says it is $8.99. I actually saw it at Sally's for $6.99 though. Wherever you get it from, I guarantee that it is worth the price.
Beauty Bit- Cantu Shea Butter Coconut Curling Cream Review
I love Cantu Shea Butter Coconut Curling Cream. It is a cheap natural hair product that works. I love how shiny and moisturized it makes my hair. I use it on dry hair in the middle of the week between washes, and I use it on wet hair as part of the LOC method to set and moisturize my twists.
Beauty Bit- Blessed Inspirations Product Review
Today I wanted to review three products from Blessed Inspirations. They are a beauty company that specializes in natural beauty products for hair and body. Last month I was given some beauty products to try from Blessed Inspirations when I went to the Charleston Natural Hair Expo. All of the products I tried at their booth were amazing, but the products I brought home were even better. Their Willow Bark extract face soap, Peppermint-Rosemary hair oil, and Lip Balm are really good products with all natural ingredients. The prices aren't bad. If you are looking for natural products, this company is one to try. I highly recommend them. The next product I want to buy is their body oil. I tried it at the hair expo, but they only had one bottle left and it was the tester.
You can buy their products at http://www.yourblessedinspiration.com and don't forget about the $140 J. Crew Gift Card Giveaway!
FCC: I was given these items by the company to try. This is an honest review
Purgasm Shop Product Launch Event!!
Purgasm Shop is a local brand of natural products for skin and hair, and I can't get enough of them. My favorite is the whipped butters, but I am fond of the scrubs and lather bars too! If you are in town, make sure to rsvp and stop by for this event. I'm sure it is going to be a blast!
Beauty Bit- Curls Unleashed Product Review
Here is a review of some of the Curls Unleashed products that I either bought myself or got as a sample at the Curls Unleashed booth at Fro Fashion Week! I think these products are great. I just think I should have used the curling jelly/cream separately instead of together. My 4a hair can't take that. The Curls Unleashed Moisturizing Conditioner is my fave product. I use it to co-wash with every week because it leaves my hair very soft and it is great for detangling my hair as well. You can find them at any local drugstore, beauty supply store, and Sally's/Walmart/Target chain. All the products are less than $15.
FCC: I paid for some of these products myself and also received some as free samples. This is an honest review.
Beauty Bit- Design Essentials Natural Line
Hey everybody! I tried these products from the Design Essentials Natural line that I received in a gift bag at Fro Fashion Week, and while I think they work great for some, they didn't work that well for me. That is why you have to try hair products to see what works. Just because this line works well with most of the people you know, doesn't mean it will work well for you. It was horrible on my hair, and was quite disappointed because I had heard such wonderful things about these products. Don't let this review scare you though. Your results might be very different from my own. I think I will eventually try it again in the future.
You can find the Design Essentials Natural line at Walgreens, Walmart, Sally's, beauty supply stores, and other drug/beauty stores.
You can also find it here:http://www.designessentials.com/natural
Etsy Love- Bel Nouvo Avocado and Shea Healthy Hair Milk
I hope you all had a safe and pleasant holiday weekend! So umm who knew Etsy sold beauty products beyond soaps? Well I just did when I went looking to restock my Bel Nouvo Avocado and Shea Healthy Hair Milk. I am in love with the Bel Nouvo Avocado and Shea Healthy Hair Milk! It has worked wonders on my hair and I can't say enough crazy good things about it. I received a tiny little sample at Fro Fashion Week in February. Since then I have been using it every now and then when I feel I need moisture or wash my hair.
I am almost out of the bottle and I need a new one, so I decided I was going to buy a full size bottle. Since I can't find this brand anywhere in my neck of the woods, I decided to look online. I was able to find the leave-in conditioner on Etsy for $10! I can't wait to buy some because it is so good. It doesn't coat my hair or make it feel heavy, you only need a little bit of this thick and very creamy stuff for each section of hair. Plus it has so many uses: Hair Moisturizer, Leave-In Conditioner, and Detangler! Also be sure to check out the Bel Nouvo Facebook page for sales and other natural hair tips.
Product Info: An rich combination of avocado oil & unrefined shea butter that will leave your hair feeling hydrated, soft, and down right spoiled by this luxurious hair treat!
USE: On wet hair, use after you wash and condition your hair for superior results. On dry hair, distribute and massage evenly throughout hair & scalp. Style as you desire
Key Ingredients:
Avocado Oil, Unrefined Shea Butter, Castor Oil, Carrot Seed and Burdock Root Extract
Scent: Sweet Sugar Cane
My First Curl Box
Hey Everyone! I finally got my hands on one of those infamous Curl Boxes from curlBox! They opened up their membership one day on Twitter so I bought a one month subscription to the natural hair product club. Make sure you check out their Twitter account so you can find out when they open up their subscriptions.


text:
BAGHDAD — As workers gathered for breakfast on Thursday at a restaurant in Hilla, a Shiite-dominated city south of the capital, an explosives-laden car parked outside blew up, sending shrapnel ripping through the crowd. As an ambulance raced to the scene, another car was in pursuit, driven by a suicide bomber — who detonated his payload just as the first responders gathered.
The bombings in Hilla, in which security and hospital officials said 32 people were killed and 138 were wounded, were the worst in a day of insurgent attacks across Iraq. Other attacks in Karbala, a Shiite holy city in the south, Falluja, Mosul and Baghdad left 16 people dead and about 70 wounded.
Security was tightened in Baghdad with new checkpoints set up before the Muslim weekend, starting Friday.
As devastating as the attacks were, they were not out of the ordinary. Nearly 10 years after the American invasion, and almost a year since the departure of the American military, assassinations, roadside bombs and suicide attacks are almost daily occurrences. Each time a period of calm sets in, it is shattered by more violence.
Just this week, Iraqis were cautiously hopeful about security because last weekend an estimated two million Shiite pilgrims descended on Karbala to commemorate Ashura, one of the holiest periods on the Shiite calendar that marks the martyrdom of a revered imam, and for the first time in years they were not targeted by insurgents.
"Whenever we feel that security is getting better, there comes a shock and any stability is gone," said a taxi driver in Hilla, who declined to give his name. "These attacks will never end. The heart of Iraq must keep bleeding."Continue reading the main story
Mushrik Kareem, an 18-year-old college student, was near the restaurant in Hilla with a friend when the attack occurred. He was hit with shrapnel and lost consciousness, but he survived. His friend, however, was killed. "When I woke up in the hospital, they told me: 'Sorry, but you lost your friend. His head was blown off his body.' I can't believe it, and I never will," he said.
Hassan Kamoona, a member of the provincial council in Hilla, said the authorities had received intelligence that indicated that an attack was coming and blamed the local security forces for not taking extra precautions to prevent the bombing.
Elsewhere, a car bomb exploded near a checkpoint on the outskirts of Karbala, killing eight people. In Baghdad, two car bombs — one struck an army checkpoint and the other a small market — killed two soldiers and one civilian. In Falluja, a suicide bomber struck a checkpoint and killed three soldiers, while in Mosul two police officers were killed by a car bomb.Continue reading the main story


text:
CHRISTINA Сlinical представляет собой основанный на новейших технологиях и инновационных ингредиентах оптимальный домашний уход, предназначенный для подготовки к процедурам, реабилитации после них, а также усиления и пролонгации полученного результата.
Cosmetic products by Christina Company can effectively solve certain aesthetic skin problems:
Aptos is recognized as the most advanced and less invasive aesthetic correction method.
Pluryal from the company MD Skin Solutions (Luxembourg) - is the latest generation of fillers based on hyaluronic acid. Dermal fillers Pluryal meet all European standards, the products are certified in accordance with ISO 13485, which guarantees the efficacy, safety and resistance results.
Pluryal from Meso company MD Skin Solutions - This is reheneratyvnaya mesotherapy. The purpose of mesotherapy in recovery synthesis of collagen, elastin and hyaluronic acid.
Depileve is the world's number 1 in professional waxing systems available
Graftobian High Definition Cosmetics is the leading brand in today's TV, Film and Print Media world.
ANESI (Spain) is a range of effective programs for comprehensive face skin care and body shaping.
HYAcorp Fine (Germany) is a last-generation biorevitalizer created using native hyaluronic acid.
Alexandria Professional (USA) is the developer of sugar waxing method of hair removal ("sugaring").
American company Mei-Cha International is a recognized leader in the industry of micro pigmentation.
The product range of 3D Beauty International is represented by diverse products for professional eyelash extension.
ANESI (Spain) is a range of effective programs for comprehensive face skin care and body shaping. These programs were developed by PHYTOLAB laboratory specialists, their unique effectiveness is proven in practice by professional beauticians in many countries all over the world, and their homecare products consolidate the results of salon procedures.
The whole cosmetic line is based on complex biotechnological, botanical extracts and specific oils effectively interacting with each other and providing optimal result. The comprehensive care of ANESI is well-known as a list of unique programs for various clients of beauty salons and SPA salons, which guarantee perfection of skin of any type.
Professionals of the beauty industry have proven the effectiveness of TM ANESI products in practice. The main advantage of these products is that each of them is focused on individual approach to a particular problem. Active ingredients help improve the skin condition of any type, are effective against cellulite, swelling and fatty deposits, cope with weak circulation of blood, which affects the structure of tissues and the process of skin nutrition.
ANESI cosmetic line has been developed on the basis of the latest biotech facilities, plant oils and extracts, complementing and reinforcing each other's action. The composition of various modeling and toning creams, massage oils, peels, serums and lotions include collagen and elastin, thermally active components that provide fat burning and reduction of body volume, and also remove excess fluid.
All series have strong positive effect over common skin condition. Isotonic series, which includes a variety of creams, lotions, cleansers, serums and masks, is close to blood serum in its composition, so besides body shaping it ensures skin recovery and health boost. Moisturizing is achieved due to Hydroxan CP, plant extracts, and concentrated sea water; skin nutrition is performed due to use of oils of corn, soy, sesame; purification is performed due to ginseng, licorice and soapwort.
Lipo-reducing effect stimulates metabolism of fat cells, reduces external evidence of cellulite, and improves skin appearance and texture. The unique formulas of cosmetics for body shaping include carefully selected ingredients that tone and renew tissue elasticity, prevent stretch marks, increase elasticity and regenerative capacity of skin.