{"action":"opened","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/7351","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/7351/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/7351/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/7351/events","html_url":"https://github.com/fchollet/keras/issues/7351","id":243225158,"number":7351,"title":"GPU Utilization","user":{"login":"KlaymenGC","id":6898967,"avatar_url":"https://avatars4.githubusercontent.com/u/6898967?v=4","gravatar_id":"","url":"https://api.github.com/users/KlaymenGC","html_url":"https://github.com/KlaymenGC","followers_url":"https://api.github.com/users/KlaymenGC/followers","following_url":"https://api.github.com/users/KlaymenGC/following{/other_user}","gists_url":"https://api.github.com/users/KlaymenGC/gists{/gist_id}","starred_url":"https://api.github.com/users/KlaymenGC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KlaymenGC/subscriptions","organizations_url":"https://api.github.com/users/KlaymenGC/orgs","repos_url":"https://api.github.com/users/KlaymenGC/repos","events_url":"https://api.github.com/users/KlaymenGC/events{/privacy}","received_events_url":"https://api.github.com/users/KlaymenGC/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-07-16T08:20:26Z","updated_at":"2017-07-16T08:20:26Z","closed_at":null,"body":"Hello,\r\n\r\nI'm doing semantic segmentation using Keras and I've notice that the GPU utilization is always fluctuating between 0% and 100% but never under full load (or fluctuating within a tighter range like 80% to 100%). And sometimes for each epoch the training time varies (for example, normally it takes 1000s but sometimes it would take 3000s).\r\n\r\nI know that for semantic segmentation you have to calculate for every pixel and it's expensive, I'm using multiprocessing in fit_generator so that the GPU shouldn't theoretically always wait for the data. My questions are: \r\n\r\n1. Is the loss calculated on the CPU then the gradients are transferred back to the GPU? What improvements can be done in Keras to achieve better GPU utilization?\r\n\r\n2. Why is the training time for each epoch sometimes varies? \r\n\r\nThanks in advance!"}}
{"actor":{"display_login":"KlaymenGC"}}