AdamW implementation
Hi,I was looking at theimplementation of the AdamW optimizerand I didn’t understand why you put the weight decay at the end.Shouldn’t you swap between this line:p.data.addcdiv_(exp_avg, denom, value=-step_size)and the weight decay part?Thanks.The AdamW algorithm from the “DECOUPLED WEIGHT DECAY REGULARIZATION” paper & The relevant source code for transformers.AdamW:image1681×794 201 KB
2021-07-16T14:39:28Z
The two lines substract an independent thing to the model parameters, so executing them in any order will give the same results.
2021-07-17T10:39:48Z
Thanks for the fast reply.I’m not sure why they are independent.The weight decay line is adding the parameter multiplied by-lr*wdto the parameter itself, so it depends on the parameter’s data.The first line is adding a value to the parameter’s data i.e. changing it.So the first line is affecting the value to be added in the second line which makes them dependent.In the following imagexdenotes the parameter’s data,adenotes the value added in line 1 andbreplaceslr*wdSo the updatedxis different in the two versions.Thanks again.
2021-07-17T14:08:09Z
Ah yes you’re right, it is indeed in the wrong place and should be put first, thanks for the lengthy explanation, I was a bit too tired when I read your post firstHowever there is a line# Add weight decay at the end (fixed version)in the code introduced by@thomwolfso there is probably a reason this is put in second.
2021-07-17T18:29:50Z
Oh that’s totally finethanks for your replies!I’ll wait for@thomwolf’s response.