title: Tensorflow: _variable_with_weight_decay(...) explanation
body: <p>at the moment I'm looking at the <a href="https://www.tensorflow.org/tutorials/deep_cnn/" rel="nofollow noreferrer">cifar10 example</a> and I noticed the function _variable_with_weight_decay(...) in the file <a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10.py#L115" rel="nofollow noreferrer">cifar10.py</a>. The code is as follows:</p>
  
  <pre><code>def _variable_with_weight_decay(name, shape, stddev, wd):
    """Helper to create an initialized Variable with weight decay.
    Note that the Variable is initialized with a truncated normal distribution.
    A weight decay is added only if one is specified.
    Args:
      name: name of the variable
      shape: list of ints
      stddev: standard deviation of a truncated Gaussian
      wd: add L2Loss weight decay multiplied by this float. If None, weight
          decay is not added for this Variable.
    Returns:
      Variable Tensor
    """
    dtype = tf.float16 if FLAGS.use_fp16 else tf.float32
    var = _variable_on_cpu(
        name,
        shape,
        tf.truncated_normal_initializer(stddev=stddev, dtype=dtype))
    if wd is not None:
      weight_decay = tf.mul(tf.nn.l2_loss(var), wd, name='weight_loss')
      tf.add_to_collection('losses', weight_decay)
    return var
  </code></pre>
  
  <p>I'm wondering if this function does what it says. It is clear that when a weight decay factor is given (wd not None) the deacy value (weight_decay) is computed. But is it every applied? At the end the unmodified variable (var) is return, or am I missing something?</p>
  
  <p>Second question would be how to fix this? As I understand the value of the scalar weight_decay must be subtracted from each element in the weight matrix, but I'm unable to find a tensorflow op that can do that (adding/subtracting a single value from every element of a tensor). Is there any op like this?
  As a workaround I thought it might be possible to create a new tensor initialized with the value of weight_decay and use tf.subtract(...) to achieve the same result. Or is this the right way to go anyway?</p>
  
  <p>Thanks in advance.</p>