title: tensorflow c++ batch inference
body: <p>I have a problem with making inference on a batchsize greater than 1 using the c++ tensorflow api. The network input planes are 8x8x13 and the output is a single float. When I try to infer on multiple samples as follows, the result is correct only for the first sample. I used keras2tensorflow tool for converting the graph to .pb format. </p>
  
  <pre><code>node {
    name: "main_input"
    op: "Placeholder"
    attr {
      key: "dtype"
      value {
        type: DT_FLOAT
      }
    }
    attr {
      key: "shape"
      value {
        shape {
          dim {
            size: -1
          }
          dim {
            size: 8
          }
          dim {
            size: 8
          }
          dim {
            size: 12
          }
        }
      }
    }
  }
  </code></pre>
  
  <p><strong>Edit:</strong> Output node is a scalar. Could the culprit be the keras2tensorflow code that I used to convert keras hdf5 file to pb ? Maybe the output should have been a -1x1 to accept any number of samples just like the input planes). I got the converter code from the following link: <a href="https://github.com/bitbionic/keras-to-tensorflow/blob/master/k2tf_convert.py" rel="nofollow noreferrer">keras_to_tensorflow</a></p>
  
  <pre><code>node {
    name: "value_0"
    op: "Identity"
    input: "strided_slice"
    attr { 
      key: "T"
      value {
        type: DT_FLOAT
      }
    }
  }
  </code></pre>
  
  <p>The input plane dimension is correctly set to -1 x 8 x 8 x 13.</p>
  
  <pre><code>void test() {
  
      //input planes
      const int nmoves = pstack-&gt;count; //This is the number of samples
      TensorShape input_shape({nmoves, 8, 8, CHANNELS});
      Tensor inputs(DT_FLOAT, input_shape);
  
      //.... Initialize input planes
  
      //output
      std::vector&lt;Tensor&gt; outputs;
  
      //run session
      TF_CHECK_OK( session-&gt;Run(
          {{input_layer, inputs}}, {output_layer}, {}, &amp;outputs) 
      );
  
      //get results
      auto outd = outputs[0].flat&lt;float&gt;().data(); //is this correct way to access the data for multiple samples ?
      for(int i = 0;i &lt; nmoves; i++) {
          float p = outd[i];    //The value of p is wrong for all but the first one
          std::cout &lt;&lt; "I" &lt;&lt; i &lt;&lt; " == " &lt;&lt; p &lt;&lt; std::endl;
      }
  }
  </code></pre>
  
  <p>Example output (p) for each sample where the result is supposed to be between 0 and 1 is shown below. Only I0 is correct while I16 and I18 have very large values. <strong>I think the problem is that after running the session the dimension of outputs is still 1, should have been 20. Is it possible at all to do inference on multiple samples using the c++ api ?</strong></p>
  
  <pre><code>I0 == 0.434162
  I1 == 0
  I2 == 0
  I3 == 0.0640963
  I4 == 0.0718748
  I5 == 0.325485
  I6 == 0
  I7 == 0
  I8 == 0
  I9 == 0
  I10 == 0.141193
  I11 == 0.398055
  I12 == 0.237758
  I13 == 0.530693
  I14 == 2.44527e-42
  I15 == 0
  I16 == -5.62959e+14
  I17 == 4.56697e-41
  I18 == -5.62959e+14
  I19 == 4.56697e-41
  </code></pre>