{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14486","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14486/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14486/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14486/events","html_url":"https://github.com/tensorflow/tensorflow/issues/14486","id":273153224,"number":14486,"title":"TowerLoss(Multiple GPU) will hurt final accuracy/performance so much when doing image finetune, is it a bug?","user":{"login":"chenghuige","id":6323467,"avatar_url":"https://avatars0.githubusercontent.com/u/6323467?v=4","gravatar_id":"","url":"https://api.github.com/users/chenghuige","html_url":"https://github.com/chenghuige","followers_url":"https://api.github.com/users/chenghuige/followers","following_url":"https://api.github.com/users/chenghuige/following{/other_user}","gists_url":"https://api.github.com/users/chenghuige/gists{/gist_id}","starred_url":"https://api.github.com/users/chenghuige/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chenghuige/subscriptions","organizations_url":"https://api.github.com/users/chenghuige/orgs","repos_url":"https://api.github.com/users/chenghuige/repos","events_url":"https://api.github.com/users/chenghuige/events{/privacy}","received_events_url":"https://api.github.com/users/chenghuige/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-11-11T15:32:32Z","updated_at":"2017-11-11T18:08:43Z","closed_at":"2017-11-11T18:08:43Z","author_association":"NONE","body":"From the paper \"Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge\"\r\nWriter say \"Training was\r\ndone using a single GPU (Nvidia K20), and step time was\r\nabout 3 seconds. Thus, training took over 3 weeks – **parallelizing\r\ntraining yielded somewhat worse results,** though it\r\nincreased the speed to convergence.\"\r\nFor my applications, I find tower loss is ok when you do anything without finetune image mode, even \r\nif you use image model(inception resnet nasnet etc.) is fine.\r\nBut if you do finetune, either for image caption or image classification, the performance will hurt \r\na lot(using mulitple gpu wether increase total batch size or keep total batch size the same as single gpu), might not convergent.  Is it a known bug, can we avoid this ?\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/343683285","html_url":"https://github.com/tensorflow/tensorflow/issues/14486#issuecomment-343683285","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14486","id":343683285,"user":{"login":"bignamehyp","id":3474655,"avatar_url":"https://avatars2.githubusercontent.com/u/3474655?v=4","gravatar_id":"","url":"https://api.github.com/users/bignamehyp","html_url":"https://github.com/bignamehyp","followers_url":"https://api.github.com/users/bignamehyp/followers","following_url":"https://api.github.com/users/bignamehyp/following{/other_user}","gists_url":"https://api.github.com/users/bignamehyp/gists{/gist_id}","starred_url":"https://api.github.com/users/bignamehyp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bignamehyp/subscriptions","organizations_url":"https://api.github.com/users/bignamehyp/orgs","repos_url":"https://api.github.com/users/bignamehyp/repos","events_url":"https://api.github.com/users/bignamehyp/events{/privacy}","received_events_url":"https://api.github.com/users/bignamehyp/received_events","type":"User","site_admin":false},"created_at":"2017-11-11T18:08:43Z","updated_at":"2017-11-11T18:08:43Z","author_association":"MEMBER","body":"This question is better asked on  [StackOverflow](http://stackoverflow.com/questions/tagged/tensorflow) since it is not a  bug or feature request. There is also a larger community that reads questions there. Thanks!"}}
{"actor":{"display_login":"bignamehyp"}}