title: Tensorflow error while backpropagating through conv2d
body: <p>I'm facing a quite strange problem. I have a network which boils down to the following "central" code:</p>
  
  <pre class="lang-py prettyprint-override"><code># COSINE
  proj = tf.multiply( proj, cosine_w, name = 'cosine-weighting' )
  
  # PARKER
  proj = tf.multiply( proj, parker_w, name = 'parker-weighting' )
  
  # RAMLAK
  s = config.proj_shape
  proj = tf.reshape( proj, [ s.N, 1, s.H, s.W ] )
  proj = tf.nn.conv2d(
         input = proj,
         filter = kernel,
         strides = [ 1, 1, 1, 1 ],
         padding = 'SAME',
         data_format = 'NCHW',
         name = 'ramlak-filter'
  )
  proj = tf.reshape( proj, config.proj_shape.toNCHW() )
  
  # BACKPROJECTION
  volume = backproject(
         projections = proj,
         # other arguments, which are attrs in the user defined op
  )
  </code></pre>
  
  <p>I get some projection data in <code>proj</code>, which is a <code>N x H x W</code> tensor (where <code>N</code> is the number of projections). This data is then weighted in two stages and afterwards filtered with a 1-dimensional filter kernel. Note here, that I do not want different weights for the different projection images (the <code>N</code> dimension). Therefore I reshape <code>proj</code> to have a size of 1 in the channels dimension and "interpret" the projection images as different images from a batch. The backproject function is a custom tensorflow op implemented in c++/cuda with a registered gradient.</p>
  
  <p>Everything works well for the forward-pass. But if I try to compute the gradient w.r.t the filter kernel, e.g. by</p>
  
  <pre><code>tf.gradients( volume, kernel, volume )
  </code></pre>
  
  <p>if get the following error:</p>
  
  <pre><code>NotFoundError (see above for traceback): No algorithm without scratch worked!
   [[Node: gradients/LAReconstructor_1/LAReconstructor/ramlak-filter_grad/Conv2DBackpropFilter = Conv2DBackpropFilter[T=DT_FLOAT, _class=["loc:@LAReconstructor_1/LAReconstructor/ramlak-filter"], data_format="NCHW", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](LAReconstructor_1/LAReconstructor/Reshape, gradients/LAReconstructor_1/LAReconstructor/ramlak-filter_grad/Shape_1, gradients/LAReconstructor_1/LAReconstructor/Reshape_1_grad/Reshape)]]
  </code></pre>
  
  <p>I tried to provide some minimal example that reproduces the error, but I could not reproduce it in such a minimal example. I have checked the gradient w.r.t to <code>proj</code> and it meets my expectations.</p>
  
  <p>Does anyone have an idea what could go wrong here?</p>
  
  <p>EDIT:</p>
  
  <p>I have just found a minimal example which produces the same error:</p>
  
  <pre><code>import tensorflow as tf
  
  proj = tf.Variable( tf.random_normal([720,1,400,600], stddev = 2) )
  kernel = tf.Variable( tf.random_normal([1, 401, 1, 1], stddev = .5), trainable = True )
  proj = tf.nn.conv2d(
      input = proj,
      filter = kernel,
      strides = [ 1, 1, 1, 1 ],
      padding = 'SAME',
      data_format = 'NCHW',
      name = 'ramlak-filter'
  )
  grad = tf.gradients( proj, kernel, proj )
  
  with tf.Session() as sess:
      sess.run( tf.global_variables_initializer() )
      print( sess.run( grad ) )
  </code></pre>
  
  <p>It seems to be related to the size of <code>proj</code>. If I change that to <code>[100, 1, 400, 600]</code>, the error disappears. But I actually need this large batch size. Any ideas?</p>