title: How to properly update the weights in PyTorch?
body: <p>I'm trying to implement the gradient descent with PyTorch according to this <a href="https://i.stack.imgur.com/XXFUe.png" rel="nofollow noreferrer">schema</a> but can't figure out how to properly update the weights. It is just a toy example with 2 linear layers with 2 nodes in hidden layer and one output.</p>
  <pre><code>Learning rate = 0.05;
  target output = 1
  </code></pre>
  <p><a href="https://hmkcode.github.io/ai/backpropagation-step-by-step/" rel="nofollow noreferrer">https://hmkcode.github.io/ai/backpropagation-step-by-step/</a></p>
  <p><a href="https://i.stack.imgur.com/XXFUe.png" rel="nofollow noreferrer">Forward</a></p>
  <p><a href="https://i.stack.imgur.com/VIUvZ.png" rel="nofollow noreferrer">Backward</a></p>
  <p>My code is as following:</p>
  <pre><code>    import torch
      import torch.nn as nn
      import torch.nn.functional as F
      import torch.optim as optim
  
      class MyNet(nn.Module):
  
      def __init__(self):
           super(MyNet, self).__init__()
           self.linear1 = nn.Linear(2, 2,  bias=None)
           self.linear1.weight = torch.nn.Parameter(torch.tensor([[0.11, 0.21], [0.12, 0.08]]))
           self.linear2 = nn.Linear(2, 1,  bias=None)
           self.linear2.weight = torch.nn.Parameter(torch.tensor([[0.14, 0.15]]))
  
      def forward(self, inputs):
           out = self.linear1(inputs)
           out = self.linear2(out)
           return out
  
      losses = []
      loss_function = nn.L1Loss()
      model = MyNet()
      optimizer = optim.SGD(model.parameters(), lr=0.05)
      input = torch.tensor([2.0,3.0])
      print('weights before backpropagation = ',   list(model.parameters()))
      for epoch in range(1):
         result = model(input )
         loss = loss_function(result , torch.tensor([1.00],dtype=torch.float))
         print('result = ', result)
         print(&quot;loss = &quot;,   loss)
         model.zero_grad()
         loss.backward()
         print('gradients =', [x.grad.data  for x in model.parameters()] )
         optimizer.step()
         print('weights after backpropagation = ',   list(model.parameters())) 
  </code></pre>
  <p>The result is following :</p>
  <pre><code>    weights before backpropagation =  [Parameter containing:
      tensor([[0.1100, 0.2100],
              [0.1200, 0.0800]], requires_grad=True), Parameter containing:
      tensor([[0.1400, 0.1500]], requires_grad=True)]
  
      result =  tensor([0.1910], grad_fn=&lt;SqueezeBackward3&gt;)
      loss =  tensor(0.8090, grad_fn=&lt;L1LossBackward&gt;)
  
      gradients = [tensor([[-0.2800, -0.4200], [-0.3000, -0.4500]]), 
                   tensor([[-0.8500, -0.4800]])]
  
      weights after backpropagation =  [Parameter containing:
      tensor([[0.1240, 0.2310],
              [0.1350, 0.1025]], requires_grad=True), Parameter containing:
      tensor([[0.1825, 0.1740]], requires_grad=True)]
  </code></pre>
  <p><strong>Forward pass</strong> values:</p>
  <pre><code>2x0.11 + 3*0.21=0.85 -&gt;  
  2x0.12 + 3*0.08=0.48 -&gt; 0.85x0.14 + 0.48*0.15=0.191 -&gt; loss =0.191-1 = -0.809  
  </code></pre>
  <p><strong>Backward pass</strong>: let's calculate w5 and w6 (output node weights)</p>
  <pre><code>w = w - (prediction-target)x(gradient)x(output of previous node)x(learning rate)  
  w5= 0.14 -(0.191-1)*1*0.85*0.05= 0.14 + 0.034= 0.174  
  w6= 0.15 -(0.191-1)*1*0.48*0.05= 0.15 + 0.019= 0.169 
  </code></pre>
  <p>In my example Torch doesn't multiply the loss by derivative so we get wrong weights after updating. For the output node we got new weights w5,w6 [0.1825, 0.1740] , when it should be [0.174, 0.169]</p>
  <p>Moving backward to update the first weight of the output node (w5) we need to calculate: <code>(prediction-target)x(gradient)x(output of previous node)x(learning rate)=-0.809*1*0.85*0.05=-0.034</code>. Updated weight <code>w5 = 0.14-(-0.034)=0.174</code>. But instead pytorch calculated <code>new weight = 0.1825</code>. It forgot to multiply by <code>(prediction-target)=-0.809</code>. For the output node we got gradients -0.8500 and -0.4800. But we still need to multiply them by loss 0.809 and learning rate 0.05 before we can update the weights.</p>
  <p>What is the proper way of doing this?
  Should we pass 'loss' as an argument to <code>backward()</code> as following: <code>loss.backward(loss)</code> .</p>
  <p>That seems to fix it. But I couldn't find any example on this in documentation.</p>