title: Converting to Python scalars
body: <p>I am implementing a SARSA reinforcement learning function which chooses an action following the same current policy updates its Q-values.</p>
  <p>This throws me the following error:</p>
  <pre><code> TypeError: only size-1 arrays can be converted to Python scalars
  
   q[s, a] = q[s, a] + eta * (reward + gamma * q[s_, a_] - q[s, a]) ValueError: setting an array element with a sequence.
  </code></pre>
  <p>I am assuming there is a problem with these lines:</p>
  <pre><code>q = np.zeros((env.n_states, env.n_actions))
  </code></pre>
  <p>and</p>
  <pre><code>q[s, a] = q[s, a] + eta * (reward + gamma * q[s_, a_] - q[s, a])
  s, a = s_, a_
  </code></pre>
  <p>This is the entire method:</p>
  <pre><code>def sarsa(env, max_episodes, eta, gamma, epsilon, seed=None):
  #environments, max number of episodes, initial learning rate, discount factor, exploration factor, seed
  
  random_state = np.random.RandomState(seed)
  
  eta =np.linspace(eta, 0, max_episodes)
  epsilon = np.linspace(epsilon, 0, max_episodes)
  q = np.zeros((env.n_states, env.n_actions))
  
  rewards = np.zeros(max_episodes)
  
  for i in range(max_episodes):
      print('starting game', i)
  
  observation = env.reset();
  s = observation
  rand = np.random.random();
  
  a = maxAction(q, s)
  done = False
  epRewards = 0
  while not done:
      observation_, reward, done = env.step(a)
      s_ = observation_
      rand = np.random.random()
      a_ = maxAction(q, s)
      epRewards += reward
      q[s, a] = q[s, a] + eta * (reward + gamma * q[s_, a_] - q[s, a])
      s, a = s_, a_
      epsilon -= 2/(max_episodes)
      rewards[i] = epRewards
  
  policy = q.argmax(axis=1)
  value = q.max(axis=1)
  
  return policy, value
  </code></pre>