title: Cloud ML Engine online prediction not scaling
body: <p>I have created a Cloud ML model that serves online predictions. The model is a simple classification model based on retraining inception v3. I require my model to process many instances (cropped images in my case) in each request. The request body is formatted as follows: </p>
  
  <pre><code>instances = []
  
  count = 0
  for item in itemMap:
  
      w = itemMap[item]['width']
      h = itemMap[item]['height']
  
      x1 = itemMap[item]['x']
      y1 = itemMap[item]['y']
      x2 = x1 + w
      y2 = y1 + h
  
      crop_x1 = int(x1-w/2)
      crop_x2 = int(x2+w/2)
      crop_y1 = int(y1-h/5)
      crop_y2 = int(y2+h/5)
  
      #cropped and resized image 
      cropped_image = frame.crop((crop_x1,crop_y1,crop_x2,crop_y2)) 
      resized_crop = cropped_image.resize((299,299))
  
      buffer = BytesIO()
      resized_crop.save(buffer, format="JPEG")
      img_str = base64.b64encode(buffer.getvalue())
  
      instances.append({"key":str(count), "image_bytes": {"b64": img_str.decode('utf-8')}})
  
      count += 1
  </code></pre>
  
  <p>Here are the key parts on my request code:</p>
  
  <pre><code>def predict_json(project, model, instances, version=None):
      """Send json data to a deployed model for prediction.
  
      Args:
          project (str): project where the Cloud ML Engine Model is deployed.
          model (str): model name.
          instances ([Mapping[str: Any]]): Keys should be the names of Tensors
              your deployed model expects as inputs. Values should be datatypes
              convertible to Tensors, or (potentially nested) lists of datatypes
              convertible to tensors.
          version: str, version of the model to target.
      Returns:
          Mapping[str: any]: dictionary of prediction results defined by the
              model.
      """
      # Create the ML Engine service object.
      # To authenticate set the environment variable
      # GOOGLE_APPLICATION_CREDENTIALS=&lt;path_to_service_account_file&gt;
      service = googleapiclient.discovery.build('ml', 'v1')
      name = 'projects/{}/models/{}'.format(project, model)
  
      if version is not None:
          name += '/versions/{}'.format(version)
  
      response = service.projects().predict(
          name=name,
          body={'instances': instances}
      ).execute()
  
      if 'error' in response:
          raise RuntimeError(response['error'])
  
      return response['predictions']
  </code></pre>
  
  <p>and:</p>
  
  <pre><code>Map = []
  
  t = time.time()
  chunk = instances[0:50]
  Mapping = predict_json(project, model, chunk)
  Map.extend(Mapping)
  
  elapsed = time.time() - t
  print(elapsed)
  </code></pre>
  
  <p>The model works but is not scaling. The time takes to send the request and receive a response is near linear.</p>
  
  <p>1 instance: 1.1077377796173096 s, 
  2 instances: 1.763524055480957 s, 
  10 instances: 8.115557432174683 s, 
  50 instances: 44.36078429222107 s, </p>
  
  <p>Any more than 50 instances and I get a time out, meaning that I have to do requests in chunks.</p>
  
  <p>I want Cloud ML to scale the nodes so that the request time is near flat (one node for each instance). I have tried sending many requests one after another to see if Cloud-Ml will allocate more nodes. I also do not want to use batch prediction as the response must come in real time. </p>
  
  <p>How do I get Cloud-ML to scale the requests and reduce my inference time? </p>
  
  <p>Update (07/12/2018): I have also tried creating a version with minNodes: 5 and there was no change in request times.</p>