{"action":"created","comment":{"url":"https://api.github.com/repos/chainer/chainer/pulls/comments/141255885","pull_request_review_id":65440400,"id":141255885,"diff_hunk":"@@ -449,6 +472,22 @@ def _zero_if_none(xp, x, shape, dtype):\n     return x\n \n \n+def _get_dtype_of_tensor_descriptor(desc):\n+    cudnn_dtype, _, _, _, _, _, _, _, _ = libcudnn.getTensor4dDescriptor(","path":"chainer/functions/normalization/batch_normalization.py","position":77,"original_position":77,"commit_id":"720f1904916ff56efa4683316b49e4452862d059","original_commit_id":"720f1904916ff56efa4683316b49e4452862d059","user":{"login":"anaruse","id":20274068,"avatar_url":"https://avatars1.githubusercontent.com/u/20274068?v=4","gravatar_id":"","url":"https://api.github.com/users/anaruse","html_url":"https://github.com/anaruse","followers_url":"https://api.github.com/users/anaruse/followers","following_url":"https://api.github.com/users/anaruse/following{/other_user}","gists_url":"https://api.github.com/users/anaruse/gists{/gist_id}","starred_url":"https://api.github.com/users/anaruse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anaruse/subscriptions","organizations_url":"https://api.github.com/users/anaruse/orgs","repos_url":"https://api.github.com/users/anaruse/repos","events_url":"https://api.github.com/users/anaruse/events{/privacy}","received_events_url":"https://api.github.com/users/anaruse/received_events","type":"User","site_admin":false},"body":"Thanks, will update in that way.","created_at":"2017-09-27T06:46:14Z","updated_at":"2017-09-27T06:46:14Z","html_url":"https://github.com/chainer/chainer/pull/3386#discussion_r141255885","pull_request_url":"https://api.github.com/repos/chainer/chainer/pulls/3386","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/chainer/chainer/pulls/comments/141255885"},"html":{"href":"https://github.com/chainer/chainer/pull/3386#discussion_r141255885"},"pull_request":{"href":"https://api.github.com/repos/chainer/chainer/pulls/3386"}},"in_reply_to_id":141219602},"pull_request":{"url":"https://api.github.com/repos/chainer/chainer/pulls/3386","id":140975453,"html_url":"https://github.com/chainer/chainer/pull/3386","diff_url":"https://github.com/chainer/chainer/pull/3386.diff","patch_url":"https://github.com/chainer/chainer/pull/3386.patch","issue_url":"https://api.github.com/repos/chainer/chainer/issues/3386","number":3386,"state":"open","locked":false,"title":"Use cuDNN BN functions even when inputs data type is fp16","user":{"login":"anaruse","id":20274068,"avatar_url":"https://avatars1.githubusercontent.com/u/20274068?v=4","gravatar_id":"","url":"https://api.github.com/users/anaruse","html_url":"https://github.com/anaruse","followers_url":"https://api.github.com/users/anaruse/followers","following_url":"https://api.github.com/users/anaruse/following{/other_user}","gists_url":"https://api.github.com/users/anaruse/gists{/gist_id}","starred_url":"https://api.github.com/users/anaruse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anaruse/subscriptions","organizations_url":"https://api.github.com/users/anaruse/orgs","repos_url":"https://api.github.com/users/anaruse/repos","events_url":"https://api.github.com/users/anaruse/events{/privacy}","received_events_url":"https://api.github.com/users/anaruse/received_events","type":"User","site_admin":false},"body":"The aim of this PR is to use cuDNN's batch-normalization (BN) functions safely when data type of input is fp16.\r\n\r\nThe cuDNN's BN functions support fp16 data type, but they are not used in Chainer now when data type of input is fp16. As far as I know, that is mainly because data types of some parameters like `gamma` and `beta` in BN layer must be fp32 in cuDNN's BN functions when data type of inputs is fp16 though Chainer assumes data types of inputs and parameters are the same.\r\n\r\nActually I've already sent a PR about this issue before (#2299). The solution of the previous PR is to decide what data type should be used for parameters like `gamma` and `beta` at the `Link` object, and keep the variables with the decided data type. With the previous PR, in case of fp16 input, the parameters data type become fp32 when cuDNN is selected to use, otherwise fp16. However there are mainly two problems:\r\n\r\n- It is less flexible. For example, a use case like using cuDNN for forward and backward and using CuPY for backward of backward is not available with the previous solution.\r\n- It is not easy-to-maintain. Both the `Link` and `Function` class are modified so they should be maintained in sync.\r\n\r\nThus, I would like to propose simple solution with this PR as follows.\r\n\r\n- To keep the data type of parameters like `gamma` and `beta` the same as the data type of inputs, even when cuDNN is likely to be used. --> No modification to the `Link` class is necessary and it is easier-to-maintain.\r\n- To convert the data type of parameters from fp16 to fp32 or vice versa on demand before or after calling cuDNN's BN functions. --> It is more flexible and the use case like using cuDNN for forward and backward and using CuPY for backward of backward gets available.\r\n\r\nPlease note that https://github.com/cupy/cupy/pull/492 needs to be used in order to test this PR.","created_at":"2017-09-14T03:28:28Z","updated_at":"2017-09-27T06:46:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8045ec49663b606a2bc1c80c85046067db40d5c8","assignee":{"login":"unnonouno","id":740680,"avatar_url":"https://avatars1.githubusercontent.com/u/740680?v=4","gravatar_id":"","url":"https://api.github.com/users/unnonouno","html_url":"https://github.com/unnonouno","followers_url":"https://api.github.com/users/unnonouno/followers","following_url":"https://api.github.com/users/unnonouno/following{/other_user}","gists_url":"https://api.github.com/users/unnonouno/gists{/gist_id}","starred_url":"https://api.github.com/users/unnonouno/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unnonouno/subscriptions","organizations_url":"https://api.github.com/users/unnonouno/orgs","repos_url":"https://api.github.com/users/unnonouno/repos","events_url":"https://api.github.com/users/unnonouno/events{/privacy}","received_events_url":"https://api.github.com/users/unnonouno/received_events","type":"User","site_admin":false},"assignees":[{"login":"unnonouno","id":740680,"avatar_url":"https://avatars1.githubusercontent.com/u/740680?v=4","gravatar_id":"","url":"https://api.github.com/users/unnonouno","html_url":"https://github.com/unnonouno","followers_url":"https://api.github.com/users/unnonouno/followers","following_url":"https://api.github.com/users/unnonouno/following{/other_user}","gists_url":"https://api.github.com/users/unnonouno/gists{/gist_id}","starred_url":"https://api.github.com/users/unnonouno/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/unnonouno/subscriptions","organizations_url":"https://api.github.com/users/unnonouno/orgs","repos_url":"https://api.github.com/users/unnonouno/repos","events_url":"https://api.github.com/users/unnonouno/events{/privacy}","received_events_url":"https://api.github.com/users/unnonouno/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/chainer/chainer/pulls/3386/commits","review_comments_url":"https://api.github.com/repos/chainer/chainer/pulls/3386/comments","review_comment_url":"https://api.github.com/repos/chainer/chainer/pulls/comments{/number}","comments_url":"https://api.github.com/repos/chainer/chainer/issues/3386/comments","statuses_url":"https://api.github.com/repos/chainer/chainer/statuses/720f1904916ff56efa4683316b49e4452862d059","head":{"label":"anaruse:BN_fp16_cudnn","ref":"BN_fp16_cudnn","sha":"720f1904916ff56efa4683316b49e4452862d059","user":{"login":"anaruse","id":20274068,"avatar_url":"https://avatars1.githubusercontent.com/u/20274068?v=4","gravatar_id":"","url":"https://api.github.com/users/anaruse","html_url":"https://github.com/anaruse","followers_url":"https://api.github.com/users/anaruse/followers","following_url":"https://api.github.com/users/anaruse/following{/other_user}","gists_url":"https://api.github.com/users/anaruse/gists{/gist_id}","starred_url":"https://api.github.com/users/anaruse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anaruse/subscriptions","organizations_url":"https://api.github.com/users/anaruse/orgs","repos_url":"https://api.github.com/users/anaruse/repos","events_url":"https://api.github.com/users/anaruse/events{/privacy}","received_events_url":"https://api.github.com/users/anaruse/received_events","type":"User","site_admin":false},"repo":{"id":63754829,"name":"chainer","full_name":"anaruse/chainer","owner":{"login":"anaruse","id":20274068,"avatar_url":"https://avatars1.githubusercontent.com/u/20274068?v=4","gravatar_id":"","url":"https://api.github.com/users/anaruse","html_url":"https://github.com/anaruse","followers_url":"https://api.github.com/users/anaruse/followers","following_url":"https://api.github.com/users/anaruse/following{/other_user}","gists_url":"https://api.github.com/users/anaruse/gists{/gist_id}","starred_url":"https://api.github.com/users/anaruse/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anaruse/subscriptions","organizations_url":"https://api.github.com/users/anaruse/orgs","repos_url":"https://api.github.com/users/anaruse/repos","events_url":"https://api.github.com/users/anaruse/events{/privacy}","received_events_url":"https://api.github.com/users/anaruse/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/anaruse/chainer","description":"A flexible framework of neural networks for deep learning","fork":true,"url":"https://api.github.com/repos/anaruse/chainer","forks_url":"https://api.github.com/repos/anaruse/chainer/forks","keys_url":"https://api.github.com/repos/anaruse/chainer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/anaruse/chainer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/anaruse/chainer/teams","hooks_url":"https://api.github.com/repos/anaruse/chainer/hooks","issue_events_url":"https://api.github.com/repos/anaruse/chainer/issues/events{/number}","events_url":"https://api.github.com/repos/anaruse/chainer/events","assignees_url":"https://api.github.com/repos/anaruse/chainer/assignees{/user}","branches_url":"https://api.github.com/repos/anaruse/chainer/branches{/branch}","tags_url":"https://api.github.com/repos/anaruse/chainer/tags","blobs_url":"https://api.github.com/repos/anaruse/chainer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/anaruse/chainer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/anaruse/chainer/git/refs{/sha}","trees_url":"https://api.github.com/repos/anaruse/chainer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/anaruse/chainer/statuses/{sha}","languages_url":"https://api.github.com/repos/anaruse/chainer/languages","stargazers_url":"https://api.github.com/repos/anaruse/chainer/stargazers","contributors_url":"https://api.github.com/repos/anaruse/chainer/contributors","subscribers_url":"https://api.github.com/repos/anaruse/chainer/subscribers","subscription_url":"https://api.github.com/repos/anaruse/chainer/subscription","commits_url":"https://api.github.com/repos/anaruse/chainer/commits{/sha}","git_commits_url":"https://api.github.com/repos/anaruse/chainer/git/commits{/sha}","comments_url":"https://api.github.com/repos/anaruse/chainer/comments{/number}","issue_comment_url":"https://api.github.com/repos/anaruse/chainer/issues/comments{/number}","contents_url":"https://api.github.com/repos/anaruse/chainer/contents/{+path}","compare_url":"https://api.github.com/repos/anaruse/chainer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/anaruse/chainer/merges","archive_url":"https://api.github.com/repos/anaruse/chainer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/anaruse/chainer/downloads","issues_url":"https://api.github.com/repos/anaruse/chainer/issues{/number}","pulls_url":"https://api.github.com/repos/anaruse/chainer/pulls{/number}","milestones_url":"https://api.github.com/repos/anaruse/chainer/milestones{/number}","notifications_url":"https://api.github.com/repos/anaruse/chainer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/anaruse/chainer/labels{/name}","releases_url":"https://api.github.com/repos/anaruse/chainer/releases{/id}","deployments_url":"https://api.github.com/repos/anaruse/chainer/deployments","created_at":"2016-07-20T06:06:21Z","updated_at":"2017-08-01T23:06:17Z","pushed_at":"2017-09-25T00:13:53Z","git_url":"git://github.com/anaruse/chainer.git","ssh_url":"git@github.com:anaruse/chainer.git","clone_url":"https://github.com/anaruse/chainer.git","svn_url":"https://github.com/anaruse/chainer","homepage":"http://chainer.org","size":22297,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":3,"mirror_url":null,"open_issues_count":0,"forks":3,"open_issues":0,"watchers":2,"default_branch":"master"}},"base":{"label":"chainer:master","ref":"master","sha":"72c612544bec833a320141837450a3e2e0f99cfd","user":{"login":"chainer","id":23183003,"avatar_url":"https://avatars3.githubusercontent.com/u/23183003?v=4","gravatar_id":"","url":"https://api.github.com/users/chainer","html_url":"https://github.com/chainer","followers_url":"https://api.github.com/users/chainer/followers","following_url":"https://api.github.com/users/chainer/following{/other_user}","gists_url":"https://api.github.com/users/chainer/gists{/gist_id}","starred_url":"https://api.github.com/users/chainer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chainer/subscriptions","organizations_url":"https://api.github.com/users/chainer/orgs","repos_url":"https://api.github.com/users/chainer/repos","events_url":"https://api.github.com/users/chainer/events{/privacy}","received_events_url":"https://api.github.com/users/chainer/received_events","type":"Organization","site_admin":false},"repo":{"id":36914478,"name":"chainer","full_name":"chainer/chainer","owner":{"login":"chainer","id":23183003,"avatar_url":"https://avatars3.githubusercontent.com/u/23183003?v=4","gravatar_id":"","url":"https://api.github.com/users/chainer","html_url":"https://github.com/chainer","followers_url":"https://api.github.com/users/chainer/followers","following_url":"https://api.github.com/users/chainer/following{/other_user}","gists_url":"https://api.github.com/users/chainer/gists{/gist_id}","starred_url":"https://api.github.com/users/chainer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chainer/subscriptions","organizations_url":"https://api.github.com/users/chainer/orgs","repos_url":"https://api.github.com/users/chainer/repos","events_url":"https://api.github.com/users/chainer/events{/privacy}","received_events_url":"https://api.github.com/users/chainer/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/chainer/chainer","description":"A flexible framework of neural networks for deep learning","fork":false,"url":"https://api.github.com/repos/chainer/chainer","forks_url":"https://api.github.com/repos/chainer/chainer/forks","keys_url":"https://api.github.com/repos/chainer/chainer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/chainer/chainer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/chainer/chainer/teams","hooks_url":"https://api.github.com/repos/chainer/chainer/hooks","issue_events_url":"https://api.github.com/repos/chainer/chainer/issues/events{/number}","events_url":"https://api.github.com/repos/chainer/chainer/events","assignees_url":"https://api.github.com/repos/chainer/chainer/assignees{/user}","branches_url":"https://api.github.com/repos/chainer/chainer/branches{/branch}","tags_url":"https://api.github.com/repos/chainer/chainer/tags","blobs_url":"https://api.github.com/repos/chainer/chainer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/chainer/chainer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/chainer/chainer/git/refs{/sha}","trees_url":"https://api.github.com/repos/chainer/chainer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/chainer/chainer/statuses/{sha}","languages_url":"https://api.github.com/repos/chainer/chainer/languages","stargazers_url":"https://api.github.com/repos/chainer/chainer/stargazers","contributors_url":"https://api.github.com/repos/chainer/chainer/contributors","subscribers_url":"https://api.github.com/repos/chainer/chainer/subscribers","subscription_url":"https://api.github.com/repos/chainer/chainer/subscription","commits_url":"https://api.github.com/repos/chainer/chainer/commits{/sha}","git_commits_url":"https://api.github.com/repos/chainer/chainer/git/commits{/sha}","comments_url":"https://api.github.com/repos/chainer/chainer/comments{/number}","issue_comment_url":"https://api.github.com/repos/chainer/chainer/issues/comments{/number}","contents_url":"https://api.github.com/repos/chainer/chainer/contents/{+path}","compare_url":"https://api.github.com/repos/chainer/chainer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/chainer/chainer/merges","archive_url":"https://api.github.com/repos/chainer/chainer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/chainer/chainer/downloads","issues_url":"https://api.github.com/repos/chainer/chainer/issues{/number}","pulls_url":"https://api.github.com/repos/chainer/chainer/pulls{/number}","milestones_url":"https://api.github.com/repos/chainer/chainer/milestones{/number}","notifications_url":"https://api.github.com/repos/chainer/chainer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/chainer/chainer/labels{/name}","releases_url":"https://api.github.com/repos/chainer/chainer/releases{/id}","deployments_url":"https://api.github.com/repos/chainer/chainer/deployments","created_at":"2015-06-05T05:50:37Z","updated_at":"2017-09-27T05:58:00Z","pushed_at":"2017-09-27T05:05:20Z","git_url":"git://github.com/chainer/chainer.git","ssh_url":"git@github.com:chainer/chainer.git","clone_url":"https://github.com/chainer/chainer.git","svn_url":"https://github.com/chainer/chainer","homepage":"https://chainer.org","size":17624,"stargazers_count":2965,"watchers_count":2965,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":789,"mirror_url":null,"open_issues_count":459,"forks":789,"open_issues":459,"watchers":2965,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/chainer/chainer/pulls/3386"},"html":{"href":"https://github.com/chainer/chainer/pull/3386"},"issue":{"href":"https://api.github.com/repos/chainer/chainer/issues/3386"},"comments":{"href":"https://api.github.com/repos/chainer/chainer/issues/3386/comments"},"review_comments":{"href":"https://api.github.com/repos/chainer/chainer/pulls/3386/comments"},"review_comment":{"href":"https://api.github.com/repos/chainer/chainer/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/chainer/chainer/pulls/3386/commits"},"statuses":{"href":"https://api.github.com/repos/chainer/chainer/statuses/720f1904916ff56efa4683316b49e4452862d059"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"anaruse"}}