title: Adam optimizer with warmup on PyTorch
body: <p>In the paper <a href="https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf" rel="noreferrer">Attention is all you need</a>, under section 5.3, the authors suggested to increase the learning rate linearly and then decrease proportionally to the inverse square root of steps.</p>
  <p><a href="https://i.stack.imgur.com/GQurA.png" rel="noreferrer"><img src="https://i.stack.imgur.com/GQurA.png" alt="Paper SS" /></a></p>
  <p>How do we implement this in PyTorch with <a href="https://pytorch.org/docs/stable/optim.html#torch.optim.Adam" rel="noreferrer">Adam</a> optimizer? Preferably without additional packages.</p>