title: What is the Best way to define Adam Optimizer in PyTorch?
body: <p>For most PyTorch codes we use the following definition of Adam optimizer,</p>
  <pre><code>optim = torch.optim.Adam(model.parameters(), lr=cfg['lr'], weight_decay=cfg['weight_decay'])
  </code></pre>
  <p>However, after repeated trials, I found that the following definition of Adam gives 1.5 dB higher PSNR which is huge.</p>
  <pre><code>optim = torch.optim.Adam(
              [
                  {'params': get_parameters(model, bias=False)},
                  {'params': get_parameters(model, bias=True), 'lr': cfg['lr'] * 2, 'weight_decay': 0},
              ],
              lr=cfg['lr'],
              weight_decay=cfg['weight_decay'])
  </code></pre>
  <p>The <a href="https://github.com/cydonia999/Learning_to_See_in_the_Dark_PyTorch/blob/470a6a8e9c6367d8fa88ee6d1dea211dd9fb1f81/models/lsid.py#L45" rel="nofollow noreferrer">Model</a> is a usual U-net with parameters defined in <code>init</code> and <code>forward action</code> as in any other PyTorch model.</p>
  <p>The <a href="https://github.com/cydonia999/Learning_to_See_in_the_Dark_PyTorch/blob/470a6a8e9c6367d8fa88ee6d1dea211dd9fb1f81/demo.py#L29" rel="nofollow noreferrer">get_parameters</a> is defined as below.</p>
  <pre><code>def get_parameters(model, bias=False):
      for k, m in model._modules.items():
          print(&quot;get_parameters&quot;, k, type(m), type(m).__name__, bias)
          if bias:
              if isinstance(m, nn.Conv2d):
                  yield m.bias
          else:
              if isinstance(m, nn.Conv2d) or isinstance(m, nn.ConvTranspose2d):
                  yield m.weight
  </code></pre>
  <p>Could someone explain why the latter definition is better than the previous one?</p>