{"action":"created","issue":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3944","repository_url":"https://api.github.com/repos/PaddlePaddle/Paddle","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3944/labels{/name}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3944/comments","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3944/events","html_url":"https://github.com/PaddlePaddle/Paddle/issues/3944","id":255792952,"number":3944,"title":"cost nan","user":{"login":"yinyunfeng","id":4621025,"avatar_url":"https://avatars1.githubusercontent.com/u/4621025?v=4","gravatar_id":"","url":"https://api.github.com/users/yinyunfeng","html_url":"https://github.com/yinyunfeng","followers_url":"https://api.github.com/users/yinyunfeng/followers","following_url":"https://api.github.com/users/yinyunfeng/following{/other_user}","gists_url":"https://api.github.com/users/yinyunfeng/gists{/gist_id}","starred_url":"https://api.github.com/users/yinyunfeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yinyunfeng/subscriptions","organizations_url":"https://api.github.com/users/yinyunfeng/orgs","repos_url":"https://api.github.com/users/yinyunfeng/repos","events_url":"https://api.github.com/users/yinyunfeng/events{/privacy}","received_events_url":"https://api.github.com/users/yinyunfeng/received_events","type":"User","site_admin":false},"labels":[{"id":651040732,"url":"https://api.github.com/repos/PaddlePaddle/Paddle/labels/User","name":"User","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-09-07T02:07:15Z","updated_at":"2017-09-07T06:38:41Z","closed_at":null,"author_association":"NONE","body":"部分日志如下：\r\n```text\r\nI0906 23:29:51.568229 436 TrainerInternal.cpp:165] Batch=200 samples=25600 AvgCost=63.6635 CurrentCost=63.6635 Eval: classification_error=0.895737 avg_tok_cost=6.19556 CurrentEval: classification_error=0.895737 avg_tok_cost=6.19556 \r\nI0906 23:31:15.191047 436 TrainerInternal.cpp:165] Batch=600 samples=76800 AvgCost=60.6319 CurrentCost=58.6425 Eval: classification_error=0.879287 avg_tok_cost=5.8936 CurrentEval: classification_error=0.870086 avg_tok_cost=5.70487 \r\nI0906 23:33:59.403811 436 TrainerInternal.cpp:165] Batch=1400 samples=179200 AvgCost=59.2 CurrentCost=57.9034 Eval: classification_error=0.87374 avg_tok_cost=5.75869 CurrentEval: classification_error=0.868735 avg_tok_cost=5.64384 \r\nI0906 23:36:02.482532 436 TrainerInternal.cpp:165] Batch=2000 samples=256000 AvgCost=58.5235 CurrentCost=55.9008 Eval: classification_error=0.868813 avg_tok_cost=5.69193 CurrentEval: classification_error=0.843411 avg_tok_cost=5.44115 \r\nI0907 00:05:40.296510 436 TrainerInternal.cpp:165] Batch=3000 samples=384000 AvgCost=49.0395 CurrentCost=52.1848 Eval: classification_error=0.76821 avg_tok_cost=4.77307 CurrentEval: classification_error=0.789352 avg_tok_cost=5.05272 \r\nI0907 00:07:01.491716 436 TrainerInternal.cpp:165] Batch=3400 samples=435200 AvgCost=49.3549 CurrentCost=51.3054 Eval: classification_error=0.770181 avg_tok_cost=4.80174 CurrentEval: classification_error=0.779835 avg_tok_cost=4.98198 \r\nI0907 00:08:11.360801 436 TrainerInternal.cpp:181] Pass=2 Batch=3744 samples=479211 AvgCost=49.447 Eval: classification_error=0.770379 avg_tok_cost=4.81064 \r\nI0907 00:09:17.907363 436 TrainerInternal.cpp:165] Batch=200 samples=25600 AvgCost=48.9454 CurrentCost=48.9454 Eval: classification_error=0.760728 avg_tok_cost=4.77426 CurrentEval:\r\nI0907 00:11:18.672466 436 TrainerInternal.cpp:165] Batch=800 samples=102400 AvgCost=48.5696 CurrentCost=47.9752 Eval: classification_error=0.756316 avg_tok_cost=4.72801 CurrentEval: classification_error=0.751526 avg_tok_cost=4.67438 \r\nI0907 00:14:00.206501 436 TrainerInternal.cpp:165] Batch=1600 samples=204800 AvgCost=48.1429 CurrentCost=47.7094 Eval: classification_error=0.752451 avg_tok_cost=4.68294 CurrentEval: classification_error=0.746996 avg_tok_cost=4.6202 \r\nI0907 00:14:41.224961 436 TrainerInternal.cpp:165] Batch=1800 samples=230400 AvgCost=48.0217 CurrentCost=47.0529 Eval: classification_error=0.751568 avg_tok_cost=4.67254 CurrentEval: classification_error=0.744492 avg_tok_cost=4.58911 \r\nI0907 00:15:22.376824 436 TrainerInternal.cpp:165] Batch=2000 samples=256000 AvgCost=48.0407 CurrentCost=48.2113 Eval: classification_error=0.752017 avg_tok_cost=4.67418 CurrentEval: classification_error=0.756047 avg_tok_cost=4.68898 \r\nI0907 00:53:18.883383 436 TrainerInternal.cpp:165] Batch=1600 samples=204800 AvgCost=43.1082 CurrentCost=42.9235 Eval: classification_error=0.708956 avg_tok_cost=4.1961 CurrentEval: classification_error=0.706402 avg_tok_cost=4.1717 \r\nI0907 00:55:22.138629 436 TrainerInternal.cpp:165] Batch=2200 samples=281600 AvgCost=43.0757 CurrentCost=43.0148 Eval: classification_error=0.708533 avg_tok_cost=4.19146 CurrentEval: classification_error=0.706836 avg_tok_cost=4.17578 \r\nI0907 00:56:03.599756 436 TrainerInternal.cpp:165] Batch=2400 samples=307200 AvgCost=43.0609 CurrentCost=42.8988 Eval: classification_error=0.708326 avg_tok_cost=4.1897 CurrentEval: classification_error=0.706047 avg_tok_cost=4.17034 \r\nI0907 00:56:44.480569 436 TrainerInternal.cpp:165] Batch=2600 samples=332800 AvgCost=43.0338 CurrentCost=42.7087 Eval: classification_error=0.708009 avg_tok_cost=4.18702 CurrentEval: classification_error=0.704209 avg_tok_cost=4.15484 \r\nI0907 00:57:25.511381 436 TrainerInternal.cpp:165] Batch=2800 samples=358400 AvgCost=43.0046 CurrentCost=42.6238 Eval: classification_error=0.707667 avg_tok_cost=4.18418 \r\nI0907 01:04:26.764242 436 TrainerInternal.cpp:165] Batch=1000 samples=128000 AvgCost=42.0039 CurrentCost=41.95 Eval: classification_error=0.698261 avg_tok_cost=4.08519 CurrentEval: classification_error=0.697248 avg_tok_cost=4.07822 \r\nI0907 01:05:49.205401 436 TrainerInternal.cpp:165] Batch=1400 samples=179200 AvgCost=41.9763 CurrentCost=41.9659 Eval: classification_error=0.697654 avg_tok_cost=4.08222 CurrentEval: classification_error=0.696067 avg_tok_cost=4.07533 \r\nI0907 01:06:30.325810 436 TrainerInternal.cpp:165] Batch=1600 samples=204800 AvgCost=41.9676 CurrentCost=41.9067 Eval: classification_error=0.697217 avg_tok_cost=4.08052 CurrentEval: classification_error=0.694165 avg_tok_cost=4.06865 \r\nI0907 01:10:34.105471 436 TrainerInternal.cpp:165] Batch=2800 samples=358400 AvgCost=41.81 CurrentCost=41.4581 Eval: classification_error=0.695526 avg_tok_cost=4.06773 CurrentEval: classification_error=0.692669 avg_tok_cost=4.03757 \r\nI0907 01:11:55.789259 436 TrainerInternal.cpp:165] Batch=3200 samples=409600 AvgCost=41.7786 CurrentCost=41.4795 Eval: classification_error=0.695099 avg_tok_cost=4.06414 CurrentEval: classification_error=0.69222 avg_tok_cost=4.03113 \r\nI0907 01:12:36.961596 436 TrainerInternal.cpp:165] Batch=3400 samples=435200 AvgCost=41.7471 CurrentCost=41.2432 Eval: classification_error=0.694817 avg_tok_cost=4.0615 CurrentEval: classification_error=0.690291 avg_tok_cost=4.01916 \r\nI0907 01:30:45.935098 436 TrainerInternal.cpp:165] Batch=1000 samples=128000 AvgCost=40.0093 CurrentCost=39.9084 Eval: classification_error=0.677508 avg_tok_cost=3.88878 CurrentEval: classification_error=0.67724 avg_tok_cost=3.8875 \r\nI0907 01:32:46.592892 436 TrainerInternal.cpp:165] Batch=1600 samples=204800 AvgCost=nan CurrentCost=nan Eval: classification_error=0.730546 avg_tok_cost=nan CurrentEval: classification_error=0.994943 avg_tok_cost=nan \r\n```\r\n\r\n\r\ncost从63开始一直降到39，而后增长52，进而变为nan，\r\n\r\n1.  想知道nan代表什么意思，是溢出的浮点数？还是负数？\r\n2. nan是什么导致的，某个参数爆炸导致？还是某个参数为负数导致？\r\n3. 如何打印所有参数值进行查看，使用--show_parameter_stats_period智能打印梯度的最大最小平均值，不能打印每个参数的值。\r\n"},"comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments/327701814","html_url":"https://github.com/PaddlePaddle/Paddle/issues/3944#issuecomment-327701814","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3944","id":327701814,"user":{"login":"lcy-seso","id":5842774,"avatar_url":"https://avatars2.githubusercontent.com/u/5842774?v=4","gravatar_id":"","url":"https://api.github.com/users/lcy-seso","html_url":"https://github.com/lcy-seso","followers_url":"https://api.github.com/users/lcy-seso/followers","following_url":"https://api.github.com/users/lcy-seso/following{/other_user}","gists_url":"https://api.github.com/users/lcy-seso/gists{/gist_id}","starred_url":"https://api.github.com/users/lcy-seso/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lcy-seso/subscriptions","organizations_url":"https://api.github.com/users/lcy-seso/orgs","repos_url":"https://api.github.com/users/lcy-seso/repos","events_url":"https://api.github.com/users/lcy-seso/events{/privacy}","received_events_url":"https://api.github.com/users/lcy-seso/received_events","type":"User","site_admin":false},"created_at":"2017-09-07T06:38:41Z","updated_at":"2017-09-07T06:38:41Z","author_association":"CONTRIBUTOR","body":"反向梯度倒着回传，从看`avg_abs_grad`，`_decoder_input_recurrent02@decoding_layer_group.w0` 这个参数已经开始异常。"}}
{"actor":{"display_login":"lcy-seso"}}