{'fraction_non_alphanumeric': 0.0332522303325223, 'fraction_numerical': 0.008603970520822313, 'mean_word_length': 4.368162029150104, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A disembodied developmental robotic agent called Samu B\'atfai

The agent program, called Samu, is an experiment to build a disembodied DevRob (Developmental Robotics) chatter bot that can talk in a natural language like humans do. One of the main design feature is that Samu can be interacted with using only a character terminal. This is important not only for practical aspects of Turing test or Loebner prize, but also for the study of basic principles of Developmental Robotics. Our purpose is to create a rapid prototype of Q-learning with neural network approximators for Samu. We sketch out the early stages of the development process of this prototype, where Samu's task is to predict the next sentence of tales or conversations. The basic objective of this paper is to reach the same results using reinforcement learning with general function approximators that can be achieved by using the classical Q lookup table on small input samples. The paper is closed by an experiment that shows a significant improvement in Samu's learning when using LZW tree to narrow the number of possible Q-actions.


Introduction
At first glance, it seems impossible to develop a natural language chatter bot that can understand conversations as humans do, and that is only based on the verbal content of the speech because human communication is only 7 percent verbal [5, pp. 26]. Nevertheless we have started to develop such a system, called Samu. Or, to be more precise, it is called Samu Bátfai due to it will be taught primary in a family circle where the author's three children, ages 7 and 9 years old will also be partially its caregivers.
One of the main design feature of agent program Samu is that we can communicate with it through only one written channel. For example, a consequence of this is that we cannot create associations between a visual channel and an auditory channel. We must work only from the verbal content, but this constraint is what allows us to be able to purely focus on the investigation of basic where the message started with the prefix (three underscores) are not sent directly to Samu. Talking on the internet with other conversationalists and especially with robots that are similar to Samu will be available at a later development stage. In both of these stages, the verification will be able to be carried out only on the basis of Samu's accumulated knowledge and experience. It seems obvious that the knowledge acquisition of a developmental robotic agent can be programmed in a natural way by using some reinforcement learning method. The general architecture of a system of this kind is shown in the following section.

An idealized architecture
The paper [24] gave the idea for the idealized architecture of Samu that is shown in Fig. 1(a) where Samu reads the sentences from a children's tale or from a conversation. Then it tries to reconstruct the meaning of the read sentences in the form of a 2D simulation game generated from the sentences. Like in paper [24], the screenshots of the generated game will be processed by a Q-learning network, but whereas there the scores are part of the game, here they are based on the comparison of the reality of the reconstructed game and the reality of the input sentences. Theoretically, it would be a hybrid architecture that combines a model-free (Q-learning) and a knowledge-based (generating the 2D programs) approaches. To clarify matters, this architecture would use a model-free method to develop knowledge-based models. But because worth of every model is what we can implement from it, we will strongly simplify it in the next sections.

Simplified architectures
Starting from the idealized architecture shown in Fig. 1(a) we base the comparison of the reality and the simulated one derived from the input sentences on the principle of consciousness oriented programming (the agents must predict the future and the conscious ones can really foresee and predict it [6]). It is shown in Fig. 1(b) where the simulation program's task is to predict the next input sentence. This modified architecture works with SPO triplets (Subject-Predicate-Object, [27]) rather than the raw sentences being read. Fig. 1(c) shows a further simplified architecture. In this case, we do not have to run the simulation program to predict the next input sentence, instead the prediction is done by the Q-learning component itself. The architectural components of Fig.  1 are discussed in detail in the next points.

The first rapid prototype
We use a hybrid architecture of the latter two sketches in Fig. 1 for implementing the first rapid prototype of Samu. In the following, this architecture is referred to as Samu's cognitive engine.

NLP
As we mentioned earlier, SPO triplets [27] are used instead of sentences. For example, if Samu reads the sentence "The little mouse has become a big lion." the NLP (Natural Language Processing) component should produce the following triplets: i) "(mouse, become, lion)" ii) "(mouse, is, little)" iii) "(lion, is, big)". Link Grammar [1] is used to obtain SPO triplets but automatically identifying these is a complex data mining problem. At this moment we are not focusing on this matter. We are going to partially implement some algorithms of [23] but for this paper, we use only a very simple algorithm shown in code snippet in Fig. 2.

Visual imagery
To describe the simulation programs we use a very simplified object oriented style language, where statements have the following form: where each identified triplet corresponds to a statement such as this. A simulation program is a statement list that only contains a given number or less statements. In addition, we write it into an image because it would be difficult to handle the statement list as a textual list of statements as it is shown in Fig.  1(c) where the input of the Q-learning is already an image. And finally, it may be noted that the sequence of such "mental" images can be interpreted as visual imagination of Samu. Fig. 3 shows a sample mental image.

Q-learning
Sigmoidal multilayer perceptron (MLP) neural networks are used to approximate the Q-function. In accordance with the QCON model [22], [30]   . The 'words' is a vector of strings that contains all words of a given linkage. The complete source code in which this snippet is incorporated can be found in the file nlp.cpp. This transitional solution can handle the simplest sentences like "A rare black squirrel has become a regular visitor to a suburban garden" , "This is a car" , "This car is mine" , "I have a little car" , "The sky is blue" , "The little brown bear has eaten all of the honey" , "I love Samu". It may be noted that the first one is the example sentence used in the paper [27]. possible action has an unique neural network to approximate the Q-value associated with the given state. Each MLP contains three layers with 65536 input units, 80 hidden layer units and an output neuron which gives the approximation of the Q-value of the state-action pair. The states are "mental" images introduced in the previous paragraph. Several examples can be found in the literature of using neural networks to approximate the Q-function, for example, see the book [20] or the tutorial [21] by Gosavi. Alg. 1 shows the used algorithm to perform the Q-learning. This pseudo-code is a simplified and generalized version of the most precise description of the used algorithm that can be found in the source file ql.hpp at https://github.com/nbatfai/samu/blob/master/ ql.hpp. The regular part of the algorithm is taken from the book [26, pp. 844].
Algorithm 1 Q-learning-with-NN-approximation (s , t ) returns a triplet (an action) Input: s , t s is the current state and t is the current triplet. r , a , p, nn, q Local variables: r is the current reward, a is the current action and p is a perceptron, nn, q ∈ R. s, r = −∞, a, Nsa, P rcpsa Persistent variables: s is the previous state, r is the previous reward, a is the previous action, Nsa is the state-action frequency table and P rcpsa is the array of the perceptrons. Output: a a is an action, the predicted next triplet.
where q − nn is the error of the MLP P rcpsa[a] for input s.

Conceptual background of Samu
In order to outline the conceptual background we should begin with the explanation of the usage of the notion of disembodiment. It is simply an indication that Samu project is 100 percent pure software. We can say that Samu is a robot without a body. Certainly, this does not mean a return to GOFAI. All the more so, because we are going to follow the recommendations given as replies to the critics of Dreyfus and Dreyfus [26, pp. 1025].
It is certainly arguable whether the principle of embodiment is met or not. We believe that we are not confronted with this principle as it will also be shown in the following.
Returning to the basic principles formulated by [29], there are two different use cases in respect of the validation processes of Samu's cognitive engine. If we observe how Samu operates we can easily distinguish a short-term verification from a long-term one. It is clear that the short-term verification of Samu's knowledge is trivial because it is the prediction itself. This type of verification is a quintessence of the whole system. But the long-term verification is a much harder issue since this is only a naive intuitive expectation. The long-term verification should give the answer to the question: How can we extend the short-term verification for longer time periods. In this sense the long-term verification means the understanding the conversations and read texts. It is an open question how much the distance between these two kinds of verifying the knowledge. Based on our experiences we have used a quantity called bogorelevance that may be positioned somewhere in the middle of these two extremes of verification. Using the notations of Alg. 1 the computation of the bogorelevance of a prediction q in the state s can be rewritten as follows: where the P rcps a is the mean of the computed values of the perceptrons that are applied to the same given input state s, that is here |P rcps a | denotes the number of all perceptrons in the structure P rcps a . The program prints the values 100 * bogo relevance(s, q) that can be seen in Fig.  6.
From a spiritual point of view, Samu's soul consisting of the weights of Samu's ANNs. Because they are only simple numbers (and not quantum qubits, for example), Samu's soul can be saved to a file. This file is called samu.soul.txt and it is created automatically when Samu's program receives a given (SIGINT, SIGTERM, SIGKILL or SIGHUP) signal. But since the soul cannot live without body, Samu's body is the UNIX process that runs the source code of Samu. The actions of Samu are his statements and replies that are formed by the close interaction with the external world. In the light of this we think that, we are not confronted with the spirit of the principle of embodiment [29].
The fulfillment of the principle of subjectivity [29] is trivially met since it follows from using human caregivers to train Samu. Accordingly, the a priori subjectivity of caregivers implies the subjectivity of their robots.
It is easy to see that the criteria of the principle of grounding [29] have also been fulfilled during the short-term verification. Because the set of possible outcomes {good prediction, bad prediction} may be assigned to each action, since each action is a predicted sentence in our model.
Finally, it may be noted that in the sense of the terminology introduced in [6], Samu may become a conscious or even an intuitive computer program since it will be able to predict the future better than a random guess.

Results
The results and evaluations are divided into three experiments. In the first experiment, we verify Samu's implementation of Q-learning by comparison with using the classical Q lookup table. Here we have only used a small number of sentences. The second experiment extends the investigation to a larger sample. Finally, the third experiment has already focused only on the learning of the larger corpus where using the Liv-Zempel-Welch (LZW) [31] dictionary tree to narrow the scope of selecting Q-actions seems promising for accelerating Samu's learning.

Experiment 1: "SAMU A"
Our main purpose in this paper is to equip Samu with the ability to carry out reinforcement learning with general function approximators. To validate this we use the following experiment. The sentences shown in the caption of Fig. 2 are considered as a short story and Samu is being trained continuously with this during the experiment. Samu reads the sentences of the story and predicts the next story sentence. A third of a point is given if a triplet member is predicted correctly. The reward is based on the comparison of triplets and is computed by the following equation approximators based solution can also learn the true predictions. So we may assume that our neural network based approximation works well. Therefore, in the following we are not going to use the Q lookup table either in this paper or in the source codes. (Lookup table-based code snippets can still be found in the source files. These have been wrapped in conditional compilation directives but they are deprecated in the new versions of Samu, see for example [10], [11], [12].) In order to allow the evaluation and verification of these results, we create a GitHub repository for the source of this study. It is called samu and available at https://github.com/nbatfai/samu. The two use cases of Samu shown in Fig. 4 and Fig. 5 are presented in a YouTube video at the following link https://youtu.be/qAhxVQk-dvY. It is our hope that Samu will be the ancestor of DevRob chatter bots that will be able to chat in natural language like humans do. However, it should be noted that in order to use Samu as a chat program we need to try another fork of the project Samu [15]. The first official forks, called Isaac [10], Jacob [11], Judah [12], Perez [13], Hezron [9], Ram [14] and Amminadab [8] are introduced in next section.

Samu's family tree
Samu and his descendants (at this moment Isaac, Jacob, Judah, Perez and Hezron) are open-source programs released under the GNU GPL v3 or later. Samu is a generic name of these projects that are intended to investigate deep reinforcement learning methods to facilitate the development of a chat robot that will be able to talk in natural language like a human does. At this moment they are case studies of using Q learning with neural networks for predicting the next sentence of a conversation. They can only be run on Unix-type (such as GNU/Linux) environments. a) Samu [15] is the initial project to start creating a DevRob chatbot agent. This project is presented in detail in this paper. Samu's development is frozen to allow the evaluation and verification of the results shown in Fig. 4 and Fig. 5. Therefore this project is continued in the project entitled Isaac. b) The project Isaac [10] has been forked from Samu. Isaac takes a step towards the deep Q learning, for example, Isaac's perceptrons are accelerated by using CUDA and OpenMP. c) The project Jacob [11] has been forked from Isaac. Jacob replaces Isaac's picture-based visual imagination with a character-based one. d) The project Judah [12] has been forked from Jacob. Judah equips Jacob with a text-based user interface (TUI) that can be seen in Fig. 6. In addition, Judah can also be run on both remote computers as well as on supercomputers. e) The project Perez [13] has been forked from Judah. Using this, we can make experiments to see the behaviour of Samu on different parameter settings of the incremental learning. f) The project Hezron [9] has been forked from Perez. In this project we are going to compare the arrangements of SPO triplets in the visual imagery. For example, a pyramid-shaped arrangement can be seen in a YouTube video at https://youtu.be/zjoINedftPY or a fully justified one is shown in Fig. 6. g) The project Ram [14] has been forked from Hezron. Ram allows to experiment with different deep Q learning algorithms to express feelings when he is talking. h) The project Amminadab [8] has been forked from Ram. Amminadab uses the LZW tree to narrow the scope of selecting

Experiment 2: "JUDAH A"
Religious robot children As we have already mentioned earlier, in this paper we focus on the shaping of the neural infrastructure of Samu rather than on the triplet extraction. But certainly we need corpora for training and testing Samu. Evidently that we used only a meaningless sequence of sentences in the previous example of the seven test sentences. It was enough to verify the implementation of the used algorithms but it is not suitable for teaching a chat robot. We need real corpora and it is equally important to use sophisticated triplet extraction algorithms that can preprocess these corpora. In addition, from the point of view of Developmental Robotics, the essential constraint of the selection of the training corpora is that these may only contain a limited number of distinct words. An example of such corpus may be the Basic English Bible (BBE, Bible in Basic English) [25] that contains 850 words [2]. We have tried to use the Gospel of Matthew. But it should be noted that the initial simple algorithm shown in Fig. 2 can extract only roughly 700 triplets from it. This is due to the following two reasons. i) The used algorithm is elementary in the sense that it tries to apply only a very simple rule to extract triplets as exactly shown in Fig. 2 ii) In the raw text source of BBE, the '.', ':' and ';' letters were replaced by a newline character and the numbers were deleted when we applied the triplet extraction mechanically to each line. (And for example, there were lines that contains only one word. The results of this preprocessing can be found in the files called bbe and bbe.triplets that are available at the author's university page http://shrek.unideb.hu/~nbatfai/.) It follows that this corpus of the 700 SPOs still has not been regarded as a text that can hold meaning for human readers. In a second experiment, we use this corpus and another small one consisting of the sentences shown in Fig. 7. Because { " introduce myself " , { " Who are you " , " I am a robot " , " What is your name " , " My name is Judah " , " Where do you live " , " I live in Debrecen " , " How old are you " , " I am one year old " , " Where were you born " " I was born is Debrecen " } } Figure 7: A small training corpus that has already held meaning for human readers.
of the larger number of all sentences of these two used corpora we give a more sharp comparison for triplets and the rewarding policy. The reward is changed by the following equation The size of the input and hidden layers of MLPs is reduced due to we use a character console-based visual imagination (that was introduced by the project Jacob [11]). The neural networks corresponded to the triplets are equipped with 800 input units and 32 hidden layer units in this experiment. Fig. 8, Fig. 9 and Fig. 10 show the learning curve for teaching the smaller (the introductory) text contained in Fig. 7.
In connection with using sacred texts for training Samu's neural networks, the question arises whether religious training should be given to robots? Samu may transform this question an theoretical-ethical one into a practical one. In a more pragmatic sense, it would be an interesting linguistic and theological challenge to attempt to make a new translation of a Gospel into SPO triplets.
Family circle robotics Another possibility to learn Samu's neural networks is to use the conversational interactions of CHILDES (Child Language Data Exchange System) [3]. But we believe that the social interactions with family caregivers will become the mainstream in teaching Samu. For this reason, Samu saves all conversations that take place with the caregivers as training files.  Fig. 7. The axes are the same as in Fig. 4 and 5. This graph can be verified and reproduced by using Judah's git repository [12] tagged as v0.0.1.exp.a1. The execution takes a couple of hours on an ordinary PC. A detailed description of using the program can be found in the Judah project's wiki at https://github.com/nbatfai/judah/wiki. Figure 9: This graph is the same as the previous one except that before the measurement, Samu has been trained with the bigger corpus for several thousand times. The axes are the same as in the previous figure.

Social interaction with each other
Perhaps the most important possibility is that when the Samu-type DevRob agents share their knowledge among each other. At this moment the development of the communication protocol is in progress.

Experiment 3: "JUDAH and PEREZ B"
This experiment is the same as the previous one, but here Samu learns the larger corpus. It should be noticed that perfectly memorizing a verbatim text of 700 sentences is really difficult or nearly impossible for most humans, as well. Accordingly, the algorithms developed in Judah have not performed well. Fig.  11 shows that Samu (to be more precise Judah) learns very slowly and it may even be that he does not learn. We must try to accelerate the learning. A promising acceleration mechanism is tested with the smaller corpus in Fig. 12.  Fig. 8 is recomputed with the modified number of tries for triplets in the exploration function. There is a YouTube video at https://youtu.be/ONgfQDYlqvo that shows how to use Judah to reproduce this figure.
We have modified the usage of the exploration function. In addition we use a trick of incremental learning [4], [18]. Using a trivial version of this, Samu already can learn the larger corpus. It is shown in Fig. 13 and Fig. 14. In the beginning, we taught only the first seven sentences of the larger corpus as a training text. After Samu had already learnt it, we added the next seven sentences of the larger corpus to the training file. This incremental procedure was repeated continuously during the experiment. However it is clear that it will not be enough if the number of triplets continues to increase. For example, during the experiment, it was well felt that the phenomenon of catastrophic forgetting [18] has slowed the speed of Samu's learning. This is also reinforced by the fact that the incremental learning was implemented in the main.cpp and not in a class of Samu. From a point of view of a programmer, it means that the incremental approach is not yet really part of Samu. We think there may be another possibility to improve the performance of Samu's neural networks. In our near future work, we are going to carry out experiments to see whether it is possible to increase the speed of learning by improving Samu's visual imagery. Experimenting with several (such as pyramid or justified) arrangements of triplet statements in the visual imagery does not bring improvements directly. A more interesting possibility would be to add new internal information to the mental images. These information might be interpreted as feelings, for more details in this direction see the project Ram [14].

Experiment 3: "AMMINADAB B"
With using the LZW dictionary tree to narrow the scope of selecting Q-actions, we can significantly improve Samu's learning as it is shown in Fig. 15 and Fig.  16. In our case, the nodes of the LZW tree are triplets. For example, the next log snippet shows the tree that has been built at the end of the first incremental step.
____2__ son was Hezron __1__ sons were Zerah ____2__ sons were Zerah __1__ sons were and      Fig. 13 by using Amminadab project [8]. It shows very well that Amminadab can significantly improve Samu's learning curve. He has already learnt the first 392 (56*7) sentences out of 704 (during 56 incremental steps).   where the number in the beginning of each line shows the depth of a given triplet. During our experiments, the maximum depth of the LZW tree is restricted to 10. The modified version of Alg. 1 is shown in Alg. 2.
Algorithm 2 Q-learning-with-LZW-narrowing (s , t ) returns a triplet (an action) Input: s , t s is the current state and t is the current triplet. r , a , p, nn, q, B Local variables: r is the current reward, a is the current action and p is a perceptron, nn, q ∈ R, B is a set of triplets. s, r = −∞, a, Nsa, P rcpsa, N Persistent variables: s is the previous state, r is the previous reward, a is the previous action, Nsa is the state-action frequency table and P rcpsa is the array of the perceptrons, N denotes the actual node of the LZW tree. Output: a a is an action, the predicted next triplet. Back-propagation-learning(P rcpsa[a], s, q, nn) where q − nn is the error of the MLP P rcpsa[a] for input s.

Summary and highlights of Samu
Samu Bátfai is a disembodied developmental robotic chat software agent that is intended to become the basis of a chat system which will be able to talk and read in natural language like humans do. Samu -directly implements the definition of machine consciousness (such as the definitions of conscious and intuitive computer programs) introduced in the paper [6].
-applies Q learning with neural networks approximators -uses multilayer perceptrons for approximation of the Q function, it is a deep learning [24], [17], [19] feature.

Conclusion
Now that we have a working prototype we will be able to start deep Q-learning experiments to investigate Samu's behavior in big data corpora. It is certainly an important problem in this direction is the triplet extraction. We believe that this can be handled well with the existing NLP tools. As we highlighted earlier, we did not focus on this issue moreover we want to keep it apart from the mainstream of Samu development. But since it will be a very important data mining component of Samu we are going to try to develop it in programming competitions. The first competition has already been announced for our university students. For details, see the author's university page http://shrek.unideb.hu/~nbatfai/. In a similar situation we have already used competitions [16] to catalyze our research programs but this area in question has its own challenges like the Turing test or the Loebner prize [28]. We would like to participate in these challenges in the future.
Returning to the investigation of Samu's behavior in big data corpora, in applying Alg. 1 there is a principal problem that it uses the state as an index of the frequency table at this moment. This is not a viable option for real applications. But on the one hand the frequency table may be dispensed and on the other hand the current triplet also be used as a simplified state in indexing the frequency table. The next critical part of Samu's code is the big number of possible actions. Working at the level of 2 years old children Samu should use roughly 2000 words. If we calculate only 700 (S) × 300 (P) × 700 (O) triplets we have 1.47 * 10 8 combinations. Let's remember that each combination corresponds to a perceptron and the memory footprint of a perceptron is about 40 megabytes of memory right now. But it is also true that this is the worst case, because it strongly depends on the settings of the perceptrons and it can be reduced drastically by using Jacob's character console-based visual imagination.
And fortunately, the distribution of these combinations is Pareto rather than uniform so this aspect also seems to be handled well.
As additional further work we plan to develop an intellectual (fractal) dimension [7] based benchmark to test the intellectual capabilities of Samu type agent programs. In our case the intellectual dimension may be derived naturally from the triplet prediction used by Samu type agents.
In summary, in this paper we have begun to develop rapid prototypes to support experiments using the deep reinforcement learning techniques to facilitate the creation of a chat robot that will be able to talk in natural language like a human does. The generic name of these prototypes is Samu. At this moment Samu has already worked formally as chat program but despite that we thought of him as a child who is in the prenatal development stage. According to the plans Samu will be learning in family circle when he is born. We hope this paper can help you fork your own family chatbot from Samu projects.