{'fraction_non_alphanumeric': 0.030425572886578087, 'fraction_numerical': 0.007702676680146351, 'mean_word_length': 4.956422018348624, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SARL: A Computational Reinforcement Learning Model with Selective Attention

SARL: A Computational Reinforcement Learning Model with Selective Attention Maurice Grinberg (mgrinberg@nbu.bg), Evgenia Hristova (ehristova@cogs.nbu.bg) Central and East European Center for Cognitive Science, New Bulgarian University, 21 Montevideo Street, 1618 Sofia, Bulgaria A model relating eye-movements and decision making is proposed focused on the iterated prisoner’s dilemma game. Its main aim is to model previous experiments with eye-tracking recordings which show that participants attend to only a small part of the game payoff information. The model presented generates eye-movements based on two main mechanisms. The first takes into account the importance of the information attended with respect to the decision making process and while the second takes into account the variability of the information attended. The model is a discrete dynamical system which integrates learned selective attention with move choice. The model is found to reproduce fairly well the sensitivity to the payoff structure of the game and the attendance to payoffs found in experiments with human subjects. These results seem to be a promising first step in explaining the impact of partial and selective information acquisition in the prisoner’s dilemma. Keywords: eye-movements models; eye-tracking; selective attention; decision making; Prisoner’s Dilemma. Goals of the Present Work It is the main goal of the current work to present a model in which decision making is integrated with information acquisition in a single integrated mechanism. The model is tested against experimental data if it is able to account for both behavioral and information acquisition data. The model we present in this paper (SARL) is based on a model proposed and used earlier (Hristova & Grinberg, 2005a) for describing playing in iterated Prisoner’s Dilemma (PD) games with different payoffs and cooperation indexes. The new model also uses the expected subjective utility framework combined with reinforcement learning; however, it also incorporates selective attention mechanisms. The name of the model SARL comes from Selective Attention and Reinforcement Learning. This work is part of a larger effort of clarifying the cognitive processes underlying PD game playing (e.g. Hristova & Grinberg, 2005b, 2008; Grinberg et al., 2005). In this series of research not only behavioral data was gathered, but also information acquisition data (using eye-tracking recordings and computerized process tracing system). The results obtained in these studies show that players do not use all the information available and that there is dependence between the playing strategy and the information acquisition patterns. The experimental results show that under the condition of playing different PD, participants sometimes miss completely the payoff structure of the game which automatically makes any model relying on full information about the payoffs useless. The model SARL has been developed to account for such situations and be able to describe and predict eye-movement and behavioural data at the same time and provide explanations about the relation between them. The Prisoner’s Dilemma Game The Prisoner’s dilemma (PD) game is one of the most extensively studied social dilemmas. PD is a two-person game. The payoff table for this game is presented in Figure 1. In PD games, the players simultaneously choose their moves – C (cooperate) or D (defect) – without knowing their opponent’s choice. In order to be a Prisoner’s dilemma game, the payoffs should satisfy the inequalities T > R > P > S and 2R > T+S. Due to the payoff structure of this game a dilemma appears – there is no obvious best move. On one hand, the D choice is dominant for both players – i.e. each player gets larger payoff by choosing D than by choosing C no matter what the other player chooses. On the other hand, the payoff for mutual defection (P) is lower than the payoff S if both players choose their dominated C strategies (R for each player). Player II Player I Abstract C D C R, R S, T D T, S P, P Figure 1: Payoff table for the PD game. In each cell the comma separated payoffs are the Player I’s and Player II’s payoffs, respectively. As PD game is used as a model for describing social dilemmas and studying the phenomena of cooperation, there is a great interest in the conditions that could promote or hinder cooperation. There are many factors, identified experimentally, that influence the cooperation rate in playing iterated PD. Among them are framing (or the way of describing the game to the participants in a experiment), players’ goals and motivation, opponent strategy, etc. (Colman, 1995; Sally, 1995). One important characteristic which accounts for the relation between payoff structure and cooperation in PD is a quantity called cooperation index (CI) which was introduced by Rapoport and Chammah (1965). It is calculated using the equation: CI = (R–P)/(T–S). CI may vary from 0 to 1 (see Figure 2) and it is positively correlated with the percentage of C choices. An advantage in using such an index for predicting cooperation is that the