title: Ray Cluster How to Access all Node Resources
body: <p>I have access to a cluster of nodes and my understanding was that once I started ray on each node with the same redis address the head node would have access to all of the resources of all of the nodes.</p>
  
  <p>main script:</p>
  
  <pre><code>export LC_ALL=en_US.utf-8
  export LANG=en_US.utf-8 # required for using python 3 with click
  source activate rllab3
  
  redis_address="$(hostname --ip-address)"
  echo $redis_address
  redis_address="$redis_address:59465"
  ~/.conda/envs/rllab3/bin/ray start --head --redis-port=59465
  
  for host in $(srun hostname | grep -v $(hostname)); do 
      ssh $host setup_node.sh $redis_address
  done
  
  python test_multi_node.py $redis_address
  </code></pre>
  
  <p><code>setup_node.sh</code> is </p>
  
  <pre><code>export LC_ALL=en_US.utf-8
  export LANG=en_US.utf-8
  
  source activate rllab3
  
  echo "redis address is $1"
  
  ~/.conda/envs/rllab3/bin/ray start --redis-address=$1
  </code></pre>
  
  <p>and</p>
  
  <p><code>test_multi_node.py</code> is </p>
  
  <pre><code>import ray
  import time
  import argparse
  
  parser = argparse.ArgumentParser(description = "ray multinode test")
  parser.add_argument("redis_address", type=str, help="ip:port")
  args = parser.parse_args()
  print("in python script redis addres is:", args.redis_address)
  
  ray.init(redis_address=args.redis_address)
  print("resources:", ray.services.check_and_update_resources(None, None, None))
  
  @ray.remote
  def f():
      time.sleep(0.01)
      return ray.services.get_node_ip_address()
  
  # Get a list of the IP addresses of the nodes that have joined the cluster.
  print(set(ray.get([f.remote() for _ in range(10000)])))
  </code></pre>
  
  <p>Ray seems to successfully start on all nodes and the python script prints out as many IP addresses as I have nodes (and they are correct). However when printing the resources it only has the resources of one node.</p>
  
  <p>How can I make ray have access to all of the resources of all of the nodes? I must have a fundamental misunderstanding because I thought the point of setting up ray on the other nodes was to give it access to all of their resources. </p>
  
  <p>According <a href="https://ray.readthedocs.io/en/latest/using-ray-on-a-cluster.html" rel="nofollow noreferrer">to this</a> ray should autodetect the resources on a new node so I don't know what's going on here.</p>