{'fraction_non_alphanumeric': 0.024961399897066392, 'fraction_numerical': 0.004460456338994682, 'mean_word_length': 4.284904805077063, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The ideal data compression and automatic discovery of hidden law using neural network

Recently machine learning using neural networks has been developed, and many new methods have been suggested. On the other hand, a system that has true versatility has not been developed, and there remain many ﬁelds in which the human brain has advantages over machine learning. We considered how the human brain recognizes events and memorizes them and succeeded to reproduce the system of the human brain on a machine learning model with a new autoencoder neural network (NN). The previous autoencoders have the problem that they cannot deﬁne well what is the features of the input data, and we need to restrict the middle layer of the autoencoder artiﬁcially. We solve this problem by deﬁning a new loss function that reﬂects the information entropy, and it enables the NN to compress the input data ideally and automatically discover the hidden law behind the input data set. The loss function used in our NN is based on the free-energy principle which is known as the uniﬁed brain theory, and our study is the ﬁrst concrete formularization of this principle. The result of this study can be applied to any kind of data analysis and also to cognitive science.


Introduction
Recently machine learning using neural network (NN) has been developed and many new methods have been suggested [1]. When we use such NN models, we still need to tune hyperparameters such as the number of nodes or the way how they are connecting each other. For example, as convolutional NN is often used for image processing [2], this is based on our knowledge that coarsegraining is effective for image data, and we need to teach the NN which pixels are adjacent in advance. One goal of machine learning is to make a system that tells the best hyperparameters without prior knowledge. The hint to this goal lies in the system of our brain. When we see an image, the human brain automatically analyzes the image and recognizes what it is, then extracts the important information. In this process, it seems that the human brain does not need prior knowledge or any kind of teacher data. If we completely reproduce the way how our brain works, it will be the first step toward the technological singularity.
The system of the human brain is considered to be formularized by the free-energy principle [3] whose statement is that the human brain works to minimize the quantity called free-energy, but a concrete formularization has not been obtained regarding how the human brain defines such a quantity. We succeeded to formularize this quantity, by focusing on how the human brain remembers and applying the free-energy principle to the system of memory. Let us make deeper consideration about how we remember. When we see or hear things, our brain automatically remembers important information and abandons trivial information. This can be interpreted as that we remember compressed data and lose some information along with this compression. Based on this understanding, we guess that the human brain works as an autoencoder [4] which compresses the data very effectively without much loss of information. However, previous autoencoders have the problem in the point that we need to restrict the middle layer based on prior knowledge, and they can not realize the best data compression if there is a lack of prior knowledge. From such consideration, we define a new loss function and a new NN model based on the free-energy principle, and as result, the NN can reproduce the system of the human brain and automatically extract the sophisticated features of the input data.
This paper is arranged as follows. In Sec. 2, we define a new autoencoder NN model which contains an input layer, encoding layers, a memory layer, decoding layers, and an output layer. In Sec. 3, we consider the compression of the data from the viewpoint of information entropy and define the loss function imposing the free-energy principle on the NN. In Sec. 4, we discuss the role of memory and explain that we can reproduce the system of the human brain by imposing the continuity of the probability distribution of the memory. In Sec. 5, we demonstrate how this NN and the loss function work considering very easy example cases. In Sec. 6, we discuss the applicability of this study and some problems remaining unsolved. In Sec. 7, we summarize our result. Figure 1: The conceptual diagram of the neural network considered in this paper. Output from the middle layer is called memory, and correspondingly the middle layer is also called the memory layer. The memory is recorded every time a new input enters.

The model of neural network
The previous autoencoder has been used to extract the features of the input data automatically, by setting the middle layer which has fewer nodes than the input layer. The problem with the autoencoder is that the number of the nodes in the middle layer is a hyperparameter which we have to choose from prior knowledge, and there is no clue whether the choice is the best even if it seems to be successful. For example, let us consider the image of handwritten digits from 0 to 9 as the input data set. Of course, the good feature of this data set is what number the image is, and it seems best to constrain the middle layer to be able to express 10 features. However, there must be other features such as beautifulness or the size of the handwritten digits, and we cannot control all the features that the input data has.
The key to solving this problem is in our brain which automatically extracts the sophisticated features of the input data without prior knowledge. To reproduce the system of our brain, we suggest a new autoencoder NN as follows. This NN has an input layer, encoding layers, a middle layer, decoding layers, and an output layer, as shown in Fig. 1. As opposed to the usual autoencoder NN, the number of nodes in the middle layer is arbitrary.
The outstanding feature of this NN model is how to deal with the output from the middle layer. To avoid confusion, we call the output from the middle layer the memory, and correspondingly we call the middle layer the memory layer in the following. This NN records the memory every time a new input enters. How the recorded memories are used will be mentioned in Sec. 3 and Sec. 4. Another feature is the shape of the output. The output predicts the probability distribution of input data under the condition of recording corresponding memory. The example for the realization of such an output will be given in Sec. 5.

The information entropy and the loss function
In this section, to achieve the ideal data compression, we discuss the actual information of events and define the loss function. Suppose that the event E happens according to the probability P (E). In this case, E has the information entropy I(E) written as Practically, we do not know the correct probability for almost all events that happen in the real world. Let us consider taking E as the input of the model defined in Sec. 2. Then, we define the memory obtained from the memory layer as M and the final output as P (X|M ), which denotes the prediction of the probability that X entered as the input under the circumstance of recording memory M . When E enters as an input in the NN, the information of E must be conserved, which means some portion of it will be embedded in the memory M and the other portion of it will be abandoned through this NN. First, let us consider the information which M has. Just as before, we can define the information entropy of M as Unlike P (E), we can estimate P (M ) by using the recorded memories as where n(M ) is the number of M recorded up to now and N is the total number of memories recorded up to now. However, if the parameter space of the memory is much larger than N , n(M ) equals 0 usually, and it becomes difficult to define P (M ) well. The prescription for this problem will be mentioned in Sec. 4. Next, let us consider the amount of information abandoned through the NN. To estimate the abandoned information, we need to know the probability distribution of input X under the condition of recording the memory M , and let us call it P (X|M ). Then the loss of information L when E enters as input is written as From the conservation of the information of E, we can write the following equation as where X| exp denotes the expectation value of X. In the following paper, we omit "| exp " unless otherwise noted. The output of this NN tries to predict the probability distribution P (X|M ), and let us call this output P(X|M ) to clarify the true probability or the prediction of the probability. It is a well-known fact that always holds for any kind of probability distribution, and combining eq. (5) and eq. (6), we get From the free-energy principle [3], we guess that our brain saves energy as much as possible to remember events, on the other hand, it tries to reduce the loss of information. This corresponds to minimizing the right-hand side of eq. (7), and we set the loss function of this NN as From eq. (7), the minimum of L(E) correspond to I(E), which means that we can estimate the actual information of E through this NN, and the NN can realize the ideal data compression for any kind of data set which is obeying the hidden law.
Here, we mention the uncertainty of the NN which minimizes this loss function. The information entropy of the memory and abandoned information entropy are inextricably linked, which means there are several types of realization to obtain the minimum of the loss function whether the NN tends to record information as much as possible or abandon information as much as possible. To control this uncertainty, we suggest two types of new loss functions as where is a very small positive constant. If we use L 1 (E), the NN tends to memorize information as much as possible, and if we use L 2 (E), the NN tends to abandon information as much as possible.
If we take too large for L 2 (E), there is a possibility to abandon even the information which should be compressed in the memory. We will discuss how these loss functions work using simple examples in Sec. 5.

The role of memory and its probability distribution
The memory for this NN is introduced to imitate the system of our brain. To grasp the role of memory, let us explain it using an example. When we see the image of an apple, our optic nerve informs all the information to our brain such as what color each pixel of the image is. However, after a few moments, we only remember the very abstract information such as "the image was an apple", although we might remember the shape or size of the apple. In other words, our brain automatically extracts the features of the image data and records only such abstract information, and other information of the image will be abandoned. The memory of this NN works similarly to our actual memory, which means the memory of the NN expresses such abstract information. Next, we will explain how the NN grasps the features of the input data and realizes such a memory. Let us consider the image data set as input, whose pixels take 0 or 1 binary value. Then the number of the cases of the input is calculated as 2 l where l is the length of the pixel data. However, the realistic input such as handwritten digits are not random binary data and there should be probability bias. The information entropy of the input E is written as where R(E) is the redundancy because of this probability bias. Taking L(E) as the loss function, it gradually decreases toward I(E), which means the NN automatically discovers the features of the input data and reduces the redundancy. We conclude that the memory is nothing but compressed data that only has abstract information.
The problem related to memory is the definition of n(M ). As we mentioned in Sec. 3, n(M ) tends to be 0 usually. For example, we consider the case that the memory layer has 100 nodes and each node of the memory layer is quantized from 1 to 10. The number of the cases of the memory is 10 100 which is enormously larger than N , then the memory recorded once will not appear twice. To solve this problem, we add the condition that a similar memory should express similar data, which means the probability distribution of the memory is continuous. Using this condition, we can estimate n(M ) as where S is the neighborhood of M and V (S) is the number of different types of memories contained in the neighborhood S. The easy way to define S is using the distance from M as where d is a radius of S and is defined as a hyperparameter. By using this approximation, the NN becomes to record a similar input as a similar memory, and some nodes in the memory layer become not to be used. As a result, the probability distribution of the memory becomes to have some peaks on the parameter space of the memory, and this is nothing but the features of the input data. We show the conceptual diagram for the probability distribution of the memory in Fig. 2.
We guess that this is exactly the way how the human brain remembers things. The memory is a very versatile object which can express abstract information. We will discuss the application of such "memory-type object" in Sec. 6.

Demonstration for binary data sets
In this section, we demonstrate the example cases for the image data considering pixel data. Suppose that the pixel data has size l and each pixel has the binary value 0 or 1. The size and numbers of the encoding layers and decoding layers are arbitrary if they are large enough to express the hidden law. To make the discussion simple, we set the size of the memory layer to be l and each node takes the binary value 0 or 1. We set the size of the output layer also to be l, and i-th node of the output predicts the probability p i which corresponds to the probability for the i-th node of the input to have the value 1. In the following, we consider two patterns of input data set, which are completely random data set and sample data set for l = 2.

Infinite number of completely random data set
There are 2 l patterns for the random binary data of length l, and it means that the probability to choose a specific binary data is 2 −l . Taking one random data E R , the information entropy for E R is written as Next, we discuss when the loss function takes its minimum. Let us consider L 1 (E) first. In this case, recording all the information of the data E R into the memory is the best way, and this can be achieved as follows. Since the memory layer has l nodes and each node can take the binary number 0 or 1, it can express 2 l different types of memories. By taking one-to-one correspondence of 2 l input patterns and 2 l memory patterns, the complete record of input information can be achieved. Let us call the corresponding memory M R . Then the NN decode the memory M R inversely, and the output can predict input data perfectly, which means that p i = 0 for i such that the value of i-th input pixel is 0 and p j = 1 for j such that the value of j-th input pixel is 1. P(E R |M R ) is calculated as where {a} is the set of the index where the input data takes the value 1 and {b} is the set of the index where the input data takes the value 0.
Next, we calculate P (M R ). If N 2 l holds, every data set appears N/2 l times, and n(X) = N/2 l holds for any memory X because we take one-to-one correspondence of the input data and memory. From eq. (3), the probability of the memory can be written as Then the loss function is calculated as This value equals I(E R ) calculated in eq. (13), which means that the loss function cannot be reduced anymore.
Next, let us consider L 2 (E). In this case, recording no information of the data E R into the memory is the best way, and this can be achieved as follows. Recording no information is equivalent to any input data being encoded into a single type of memory, which means P (M R ) = 1. The final output is decoded from the same memory, and it always predicts the same probability distribution for any input data. It is easy to show that the loss function takes a minimum value when p i = 1/2 holds for any i. The explicit calculation of the loss function is This loss function also equals I(E R ), and we cannot reduce the loss function anymore.

Sample data set for l = 2
To help understand the essence of this theory, we will give a more concrete example for l = 2. In this case, there are 4 types of input and memory, and we denote each input and each memory as Then, for instance, we set their probabilities as In this case, the expectation value of the information entropy of input data is calculated as Let us take L 1 (E) as the loss function. The minimum of the loss function is easily achieved when the encoding layers work as and the decoding layers work as It is easy to check L 1 (E)| exp = I(E)| exp is satisfied for this NN.
Next, let us take L 2 (E) as the loss function. First, we will take a look at what will happen if the NN records no information. This situation is achieved when the encoding layers work as To make the loss function small, it is best for the output to obey the total probability for each input pixel to have the value 1, which means that the decoding layers work as M 1 → (p 1 , p 2 ) = (0.7, 0.6) .
Then the expectation value of L 2 (E) is calculated as For this NN, L 2 (E)| exp is larger than I(E)| exp , because we lose the information of the correlation between two pixels. We need to encode the information of this correlation into the memory to get the minimum of L 2 (E), and it is achieved when the encoding layers work as and the decoding layers work as which equals to I(E)| exp when is very small. From these observations, we summarize the properties of L 1 (E) and L 2 (E). The redundancy of the input information comes from the correlation between pixels or the prior probability distribution of the input. In principle, we cannot distinguish them because they are just different perspectives of the same redundancy. If we use L 1 (E) as the loss function, the NN takes the redundancy as the prior probability distribution of input and records as much information as possible. If we use L 2 (E) as the loss function, the NN takes the redundancy as the correlation between the pixels, and it records only the information of the correlation and abandons the remaining information caused by a prior probability distribution for each pixel.

Discussion
When we apply this NN to realistic data set, we have to consider how to determine the neighborhood S defined in Sec. 4. As learning of the NN proceeds, the distribution of the memory becomes clear, and so we should reduce the volume of S. This problem is likely to be solved by changing S to contain a certain constant number of memories. Another problem is how to deal with the memories recorded a long time ago. Such memories are recorded before the learning proceeds, and they do not follow the ideal memory distribution which we want to obtain. To increase the convergence of the loss function, we should abandon these memories at some moment. This problem remains to be solved. This NN is invented to make NN closer to the way how the human brain remembers events and discovers hidden laws, and as a result, the NN also reflects some important properties that the human brain has. For example, let us consider the learning of the image of animals. At first, the NN does not know what the animals are and there is no classification. As the learning proceeds, the NN gradually learns what the animals are and becomes to distinguish the animals roughly such as mammals, birds, and fish. This distinction is realized as the peaks of the probability distribution of memory. As the learning proceeds more, the NN becomes to distinguish the minor difference such as cats and dogs, which means more detailed peaks will appear in the probability distribution of memory. From such consideration, we conclude that the NN automatically compresses the information of data as much as possible and records similar data as a similar memory as the human brain does. Moreover, as we discussed the usefulness of abandoning old memories in the paragraph above, it seems to be also a common point with the human brain which tends to forget old memories that are not referred to.
Next, we discuss the usefulness of the memory-type object as mentioned in Sec. 4. The memory is compressed data containing the sophisticated features of the input data and can express abstract information such as "a big dog" or "a red apple". We can construct a new NN model whose input is the memory-type object, and then the NN can learn more complicated law. For example, let us consider entering the audio data of spoken language and image data of animals as input to the NN. The NN will automatically learn the distinction between words and the distinction between the animals. Then, if we enter the obtained memory-type objects to a new NN simultaneously, it automatically learns the combination of the word and the animal that often enter together, and it starts to learn the meaning of the words such as "this animal is called a cat". If we repeat this process, the NN will learn the correlation between more abstract information, and this can be the solution to the binding problem [5].
Finally, we briefly discuss how the human brain decides the action. When we experience some situation, our brain automatically analyzes the situation and recognizes it as a memory-type object. The human brain also defines one's action as a memory-type object, and reinforcement learning based on such memory-type objects is what the human brain is performing.

Conclusion
In this paper, we presented an autoencoder NN model with a new loss function that is based on information entropy. The previous autoencoder models have the problem of how to define the feature of the input data set, and we have to teach the autoencoder how many features in the input data set as the constraint to the middle layer. As opposed to them, our NN automatically defines the sophisticated features of the input data and realizes the best encoding and decoding of the input data. This is because the minimum of the loss function corresponds to the actual information entropy of the input data, then the NN can achieve the ideal data compression and automatic discovery of hidden low behind the input data.
This NN model is inspired by the mechanism of the human brain, and the loss function defined here is also the formularization of the free-energy principle. As discussed in Sec. 4, by setting the condition that the probability distribution of the memory is continuous, this NN automatically reduces the number of nodes used in the memory, and the distribution of the memory tends to have some peaks which correspond to the features of the input data. This situation is the same as what happens in the human brain, and the important point is that such properties automatically appear only by imposing the free-energy principle and the continuity of the memory distribution.
This NN realizes the best data compression and defines the sophisticated features for any kind of data set. This study can be applied to very wide fields which include image processing, audio processing, statistics, linguistics, and cognitive science.