{'fraction_non_alphanumeric': 0.029695862158987077, 'fraction_numerical': 0.0029369534003393814, 'mean_word_length': 4.425991501416431, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Optimization of Real, Hermitian Quadratic Forms: Real, Complex Hopfield-Amari Neural Network

In this research paper, the problem of optimization of quadratic forms associated with the dynamics of Hopfield-Amari neural network is considered. An elegant (and short) proof of the states at which local/global minima of quadratic form are attained is provided. A theorem associated with local/global minimization of quadratic energy function using the Hopfield-Amari neural network is discussed. The results are generalized to a"Complex Hopfield neural network"dynamics over the complex hypercube (using a"complex signum function"). It is also reasoned through two theorems that there is no loss of generality in assuming the threshold vector to be a zero vector in the case of real as well as a"Complex Hopfield neural network". Some structured quadratic forms like Toeplitz form and Complex Toeplitz form are discussed.


INTRODUCTION
In natural, engineering and other sciences as well as technology dynamical systems are utilized for modeling the associated dynamic phenomena. Large body of literature is developed for analyzing and synthesizing linear as well as nonlinear dynamical systems. Amari as well as Hopfield successfully discovered as well as formalized an artificial neural network modeling an associative memory. In the following the operation of such a neural network is briefly discussed for completeness: This neural network can be represented as N(W,T) where W signifies the synaptic weight matrix and T the threshold vector. The order of such a network is defined as the number of nodes in the network and denoted by n. T is the threshold vector consisting of the individual thresholds attached to each of the nodes of the neural network, represents the synaptic weight from the node to the node and represents the state of node i at time t. The network can be operated in two modes-If the state computation is done at one particular node at any instant of time, then the network is said to be in serial mode of operation and if the state computation is done at all the nodes at any instant of time, then the network is said to be in fully parallel mode of operation. The consequent state computation of any arbitrary node (i) is as follows: A state is said to be stable if and only if . We define an anti-stable state as follows-A state is said to be anti-stable if and only if .
There is a convergence theorem given by Hopfield associated with local/global maximization of quadratic energy function and is as follows-Let N=(W, T) be a neural network, with W being a synaptic weight matrix which is a symmetric matrix with nonnegative diagonal elements and T being the threshold vector; then the following hold.
1) If N is operating in a serial mode, then the network will always converge to a stable state.
2) If N is operating in a fully parallel mode, then the network will always converge to a stable state or to a cycle of length 2.
In contrast to the above problem, the authors naturally formulated the problem of minimization of real quadratic form over the real hypercube. The authors attempted and succeeded in providing an elegant and short proof of the condition satisfied by the local/global minimum anti-stable state on the real hypercube. This proof is provided in Section II. In this section we also discuss local/global minimization of real quadratic form based energy function over the real hypercube using the Hopfield-Amari neural network. In this section we also reason that there is no loss of generality in choosing the energy function of a Hopfield-Amari neural network as a quadratic form. In Section III minimization of quadratic forms associated with a Hermitian matrix (i.e., Hermitian forms) over the complex hypercube using the complex signum function is discussed. The elegant proof in Section II is generalized in the complex case. Local/global minimization of Hermitian form based energy function over the complex hypercube using the complex Hopfield-Amari neural network is also discussed. In Section III we also reason that there is no loss of generality in choosing the energy function of a complex Hopfield-Amari neural network as a Hermitian form. In Section IV the generalizations of the Toeplitz as well as the complex Toeplitz forms are provided.
We expect several problems in engineering, computer science, natural sciences and other areas requiring the minimization of quadratic form (real or Hermitian form) over the real /complex hypercube. One important motivation for the problem is the characterization of corner positive (corner positive definite) matrices that arise in the characterization of autocorrelation function of {+1, -1} valued random processes.
Definition: A real valued matrix B is corner positive if and only if for all lying on the symmetric, real hypercube.

A. Theorem(1) Statement
Let E be an arbitrary n × n real matrix. If u minimizes the quadratic form, subject to the constraint for 1≤i≤n (i.e., u lies on the real hypercube), then where C is the symmetric matrix with zero diagonal elements obtained from E.

B. Proof
Without loss of generality any arbitrary real matrix can be broken into a symmetric as well as an anti-symmetric matrix.

Let,
The quadratic form associated with Q is always zero where as the quadratic form associated with P can be written as If X lies on the real hypercube, . Therefore, In view of (2), without loss of generality, C may be assumed to be a symmetric matrix with zero diagonal elements obtained from E. Now we claim that a binary vector, which provides the minimum value for the quadratic form necessarily satisfies (1).
Suppose it does not. This implies that there exists a vector v, which achieves the minimum of the quadratic form and at the same time violates the condition (1). Suppose all the other components of v satisfy the condition (1)  Remark (1): It can be noted that the same proof can actually be used to prove the condition associated with maximization.

C. Theorem(2)
Now that we provided a characterization of anti-stable states (local minima of a real quadratic form over a real hypercube), we are naturally led to the question how they can be reached starting from an arbitrary initial state. The following theorem provides a method of reaching the local/global minima of a real quadratic form based energy function (with the threshold vector being non-zero) over the real hypercube using the Hopfield-Amari neural network. It is a convergence theorem in the spirit of the related theorem for Hopfield-Amari neural network [1].

Statement
Let N=(W, T) be a neural network, with W being the synaptic weight matrix which is a symmetric matrix with nonnegative diagonal elements and T being the threshold vector; then the following hold.

1)
If N is operating in a serial mode and the consequent state computation of any arbitrary node (i) is as follows: otherwise where , then the network will always converge to an anti-stable state.
2) If N is operating in a fully parallel mode and the consequent state computation of any arbitrary node (i) is similar to that of the serial mode, then the network will always converge to an anti-stable state or to a cycle of length 2.

Proof
The proof follows from considering the minimization of the energy function (Lyapunov function) defined on the dynamics of Hopfield-Amari neural network. The generalization follows from a similar argument.

D. Theorem(3)
The theorem (2) holds true even for the real quadratic form, as it can be proved that there is no loss of generality in neglecting the threshold vector.

Statement
There is no loss of generality in neglecting the threshold vector in a Hopfield-Amari neural network i.e., there is no loss of generality in choosing the energy function of a Hopfield-Amari neural network as a quadratic form.

Proof
In the following, we reason that in a Hopfield network there is no loss of generality in assuming that the threshold at each neuron is zero.

Consider a Hopfield network uniquely defined by (W,T)
where W is the synaptic weight matrix which is a symmetric matrix with nonnegative diagonal elements and T is the threshold vector. Now, we consider a new Hopfield network with a dummy node i.e., the node. Let the state of the node be fixed at '1'. Also the synaptic weights from node to the node are chosen as (i.e., the component of the vector T). The last diagonal element is suitably chosen to ensure that the state at the node is always fixed at 1. Thus the synaptic weight matrix of the resulting Hopfield network is given by-, where .
Also i.e., the threshold vector is a zero vector. It is now easy to reason that the serial and parallel modes of operation of the new Hopfield network retain the structure of stable/anti-stable states of the original network.
Thus, the energy function of the new Hopfield network is a quadratic form (since the threshold vector is zero).

E. Associative Memory Application
We know that the Hopfield-Amari neural network can be used as an associative memory. Hopfield synthesized a real valued synaptic weight matrix from the patterns to be stored in such a way that the network so obtained has these patterns as stable states. The weight matrix given by Hopfield is as follows: where S is the number of patterns to be stored, I is the identity matrix and are the real patterns (orthogonal to each other and also lying on the real hypercube) to be stored.
We synthesized a real valued synaptic weight matrix from the patterns to be stored in such a way that the network so obtained has these patterns as anti-stable states. The weight matrix is as follows: where S is the number of patterns to be stored, I is the identity matrix and are the real patterns (orthogonal to each other and also lying on the real hypercube) to be stored.

Note:
The real patterns are said to be orthogonal to each other if and .

III. OPTIMIZATION OF HERMITIAN FORMS OVER THE COMPLEX HYPERCUBE
In the theory of complex quadratic forms, there are various efforts made on studying Hermitian forms. To the best of our knowledge, no one studied Hermitian forms and their minimization over the complex hypercube. So, now in the following discussion we provide precise definition of the underlying concepts involved:

A. Complex Hypercube
Unlike the real hypercube there are 4 n points on the complex hypercube of n-dimensions as a vector of size n on the complex hypercube has each entry belonging to the set: {1+j, 1-j, -1+j, -1-j}.

B. Complex Signum Function
We define the complex signum function as follows: Sgn(a+jb) = sign(a) + j sign(b)

C. Equations
The following equations related to arbitrary complex quadratic forms (not just Hermitian forms) are here by provided for completeness. Specifically, their evaluation on the complex hypercube is one of our contributions.
Any complex valued matrix, A can be decomposed in the following manner: Where A H is the Hermitian part associated with A and A SH is the skew-Hermitian part. These parts have the following properties: Now, let us evaluate the quadratic form associated with A.
The sum, is real, since is real and is real.

1) Remark:
In the case of a complex hypercube Hence, = = 2 . Therefore, Considering the other sum, , this contains the sum of terms of the form, .
Thus the net contribution of all such terms is real. Hence the quadratic form associated with a Hermitian matrix is always real.
Note: If the analysis is carried out for a skew-Hermitian matrix in the similar way, we can conclude that the quadratic form associated with a skew-Hermitian matrix is always imaginary.

D. Theorem(4) Statement
Let E be an arbitrary n × n Hermitian matrix. If u minimizes the Hermitian form, subject to the constraint that u lies on the complex hypercube, then where C is the matrix with zero diagonal elements obtained from E.

E. Proof
In view of (3), without loss of generality, C may be assumed to be a matrix with zero diagonal elements obtained from E.
Hence, C can be written as follows, C Let us assume that there exists a vector w such that it minimizes the Hermitian form and also satisfies (4).
Defining w as and evaluating we get Also, we have If we can prove that there exists no v such that it minimizes the Hermitian form and at the same time violates the condition (4), then the theorem (4) can be proved.
We just vary the nth component of v in order to achieve this.

Let
. Now three possibilities arise for .

Let This implies, or or
Ruling out all the three cases, Case (1): .
Case (3): . Thus all the three cases are ruled out and hence there exists no v such that it minimizes the Hermitian form and at the same time violates the condition (4). Hence the theorem (4) is proved.
Remark (2): It can be noted that the same proof can actually be used to prove the condition associated with maximization.

F. Theorem(5)
Now that we provided a characterization of anti-stable states (local minima of a Hermitian form over a complex hypercube), we are naturally led to the question how they can be reached starting from an arbitrary initial state. The following theorem provides a method of reaching the local/global minima of a Hermitian form based energy function (with the threshold vector being non-zero) over the complex hypercube using the complex Hopfield-Amari neural network. It is a convergence theorem in the spirit of the related theorem for complex Hopfield-Amari neural network [2].

Statement
Let N=(W, T) be a neural network, with W being a synaptic weight matrix which is a Hermitian matrix with nonnegative diagonal elements and T being the threshold vector; then the following hold.

1)
If N is operating in a serial mode and the consequent state computation of any arbitrary node (i) is as follows: where , then the network will always converge to an anti-stable state.
2) If N is operating in a fully parallel mode and the consequent state computation of any arbitrary node (i) is similar to that of the serial mode, then the network will always converge to an anti-stable state or to a cycle of length 2.

Proof
The proof follows from considering the minimization of the energy function defined on the dynamics of complex Hopfield-Amari neural network. The generalization follows from a similar argument.

G. Theorem(6)
The theorem (5) holds true even for the Hermitian form, as it can be proved that there is no loss of generality in neglecting the threshold vector.

Statement
There is no loss of generality in neglecting the threshold vector in a complex Hopfield-Amari neural network i.e., there is no loss of generality in choosing the energy function of a complex Hopfield-Amari neural network as a Hermitian form.

Proof
In the following, we reason that in a complex Hopfield network there is no loss of generality in assuming that the threshold at each neuron is zero.
Consider a complex Hopfield network uniquely defined by (W,T) where W is the synaptic weight matrix which is a Hermitian matrix and T is the threshold vector. Now, we consider a new Hopfield network with a dummy node i.e., the node. Let the state of the node be fixed at ' '. Also the synaptic weights from node to the node ( ) are chosen in such a way that (i.e., the component of the vector T).
Let us suppose then we choose in such a way that . Thus, the setand can be solved as follows: i.e., ,