{"action":"opened","issue":{"url":"https://api.github.com/repos/facebookresearch/fairseq-py/issues/41","repository_url":"https://api.github.com/repos/facebookresearch/fairseq-py","labels_url":"https://api.github.com/repos/facebookresearch/fairseq-py/issues/41/labels{/name}","comments_url":"https://api.github.com/repos/facebookresearch/fairseq-py/issues/41/comments","events_url":"https://api.github.com/repos/facebookresearch/fairseq-py/issues/41/events","html_url":"https://github.com/facebookresearch/fairseq-py/issues/41","id":267940906,"number":41,"title":"Exploding in WMT14 en-fr","user":{"login":"Zrachel","id":4532062,"avatar_url":"https://avatars0.githubusercontent.com/u/4532062?v=4","gravatar_id":"","url":"https://api.github.com/users/Zrachel","html_url":"https://github.com/Zrachel","followers_url":"https://api.github.com/users/Zrachel/followers","following_url":"https://api.github.com/users/Zrachel/following{/other_user}","gists_url":"https://api.github.com/users/Zrachel/gists{/gist_id}","starred_url":"https://api.github.com/users/Zrachel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Zrachel/subscriptions","organizations_url":"https://api.github.com/users/Zrachel/orgs","repos_url":"https://api.github.com/users/Zrachel/repos","events_url":"https://api.github.com/users/Zrachel/events{/privacy}","received_events_url":"https://api.github.com/users/Zrachel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-10-24T08:27:46Z","updated_at":"2017-10-24T08:27:46Z","closed_at":null,"author_association":"NONE","body":"Hello.\r\nI've processed my data and set training parameters as the same in pre-trained models/wmt14.en-fr.fconv-py/README.md. However, I get\r\n\r\n```\r\n| [en] dictionary: 43881 types\r\n| [fr] dictionary: 43978 types\r\n| data-bin train 35482842 examples\r\n| data-bin valid 26663 examples\r\n| data-bin test 3003 examples\r\n| using 8 GPUs (with max tokens per GPU = 4000)\r\n| model fconv_wmt_en_fr\r\nWarning! 1 samples are either too short or too long and will be ignored, sample ids=[28743556]\r\n| epoch 001  1000 / 331737 loss=9.57 (10.94), wps=18515, wpb=31259, bsz=861, lr=1.25, clip=100%, gnorm=2.0540\r\n| epoch 001  2000 / 331737 loss=8.61 (9.91), wps=18466, wpb=31229, bsz=877, lr=1.25, clip=100%, gnorm=1.7149\r\n| epoch 001  3000 / 331737 loss=7.50 (9.23), wps=18493, wpb=31226, bsz=871, lr=1.25, clip=100%, gnorm=2.7501\r\n| epoch 001  4000 / 331737 loss=6.87 (8.75), wps=18522, wpb=31231, bsz=873, lr=1.25, clip=100%, gnorm=100615.8788\r\n| epoch 001  5000 / 331737 loss=10405.01 (136.96), wps=18532, wpb=31216, bsz=874, lr=1.25, clip=100%, gnorm=1500459828271.3960\r\n| epoch 001  6000 / 331737 loss=4773454961.36 (92926125.94), wps=18564, wpb=31213, bsz=867, lr=1.25, clip=100%, gnorm=37459419138681.4219\r\n| epoch 001  7000 / 331737 loss=7746569234820.15 (126329286789.38), wps=18577, wpb=31211, bsz=864, lr=1.25, clip=100%, gnorm=inf\r\n| epoch 001  8000 / 331737 loss=18016233617.10 (228909462625.55), wps=18562, wpb=31205, bsz=866, lr=1.25, clip=100%, gnorm=inf\r\n| epoch 001  9000 / 331737 loss=6500325670920.53 (321325856038.58), wps=18597, wpb=31214, bsz=860, lr=1.25, clip=100%, gnorm=inf\r\n| epoch 001 10000 / 331737 loss=11162501170786.86 (715142464195.40), wps=18609, wpb=31219, bsz=858, lr=1.25, clip=100%, gnorm=inf\r\n....\r\n```\r\n--------------------------ENV----------------------------\r\nP40 8cards\r\n\r\n\r\n--------------------------DATA   PREPROCESSING----------------------------\r\n1. normalize-punctuation\r\n2. tokenizer\r\n3. clean-corpus-n\r\n4. shuffle\r\n5. learn and apply bpe\r\nI've checked en-fr data corresponding relationship after preprocessing.\r\n\r\n--------------------------TRAINING　PARAMETER----------------------------\r\nfairseq_train_param=\"-s en -t fr --arch fconv_wmt_en_fr \\\r\n--dropout 0.1 --lr 1.25 --clip-norm 0.1 --max-tokens 4000 --force-anneal 32\"  \r\n\r\n\r\n\r\nCan you help me figure out where is my problem? Thank you."}}
{"actor":{"display_login":"Zrachel"}}