{"action":"created","issue":{"url":"https://api.github.com/repos/uber/horovod/issues/60","repository_url":"https://api.github.com/repos/uber/horovod","labels_url":"https://api.github.com/repos/uber/horovod/issues/60/labels{/name}","comments_url":"https://api.github.com/repos/uber/horovod/issues/60/comments","events_url":"https://api.github.com/repos/uber/horovod/issues/60/events","html_url":"https://github.com/uber/horovod/issues/60","id":269275136,"number":60,"title":"confirming installation and setup running examples in the code","user":{"login":"atinsood","id":515920,"avatar_url":"https://avatars0.githubusercontent.com/u/515920?v=4","gravatar_id":"","url":"https://api.github.com/users/atinsood","html_url":"https://github.com/atinsood","followers_url":"https://api.github.com/users/atinsood/followers","following_url":"https://api.github.com/users/atinsood/following{/other_user}","gists_url":"https://api.github.com/users/atinsood/gists{/gist_id}","starred_url":"https://api.github.com/users/atinsood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atinsood/subscriptions","organizations_url":"https://api.github.com/users/atinsood/orgs","repos_url":"https://api.github.com/users/atinsood/repos","events_url":"https://api.github.com/users/atinsood/events{/privacy}","received_events_url":"https://api.github.com/users/atinsood/received_events","type":"User","site_admin":false},"labels":[{"id":665452437,"url":"https://api.github.com/repos/uber/horovod/labels/question","name":"question","color":"cc317c","default":true},{"id":672824293,"url":"https://api.github.com/repos/uber/horovod/labels/tracking","name":"tracking","color":"c2e0c6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":44,"created_at":"2017-10-28T01:54:05Z","updated_at":"2017-12-15T20:20:23Z","closed_at":null,"author_association":"NONE","body":"Guys thanks for this interesting project. Appreciate the effort that you have put in here. I am trying out a few things but have a few doubts if I am doing things right.\r\n\r\nI have 2 machines, each with 2 P100s and 16gig mem on each GPU\r\nI am starting with examples/  , and executing the command `~/.openmpi/bin/mpirun  -np 4 -v -hostfile ~/hostfile -x NCCL_DEBUG=INFO ~/envs/horovod/bin/python tensorflow_word2vec.py`\r\n\r\nI can confirm that this kicks off the example across both my machines in the host files and across the total 4 gpus. \r\n\r\nBut I have a few doubts if I am missing something, as I am seeing a higher time periods of exec when running `a` single GPU with horovd as compared to vanilla single core TF `b` distributed horovod vs single GPU horovod execution\r\n\r\nSo just wanted to confirm a few things:\r\n\r\n- whether the GPUs are sufficiently being utilized \r\n\r\n(nvidia-smi seems to point out that each of the GPU is running the code and 7% of the memory is being used). \r\nFrom top:\r\n```%Cpu(s):  3.1 us,  0.8 sy,  0.0 ni, 96.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st```\r\n```\r\n PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND\r\n 20   0 29.199g 1.487g 421128 S 116.9  2.4   5:32.41 python\r\n 20   0 29.199g 1.495g 421172 S 114.6  2.4   5:24.61 python\r\n```\r\n- whether the GPUs are efficiently communicating with each other\r\n\r\n( as pointed out in the other ticket I should see something in the logs about it, but I don't see anything. I can see the processes started across machines, but I am not sure if all the communication is good. The training gets to completion, if I specify incorrect host then I do see errors, so I have a feeling that all the instances are communicating but `a` not sure if they are communicating properly `b` they are communicating via libnccl2\r\n\r\n- whether anything special needs to be done from open mpi side\r\nI basically downloaded openmpi, did not do anything special to build it optimally for say GPUs or libnccl. Do I need to anything additional to optimize/leverage for GPUs \r\n\r\n- training times are increasing as we add more GPUs\r\n\r\nvanilla tensor flow for above example ~ 3 sec\r\nhorovod single GPU ~ 4 sec\r\nhorovod with 2 GPU on same machine ~9 sec"},"comment":{"url":"https://api.github.com/repos/uber/horovod/issues/comments/352102217","html_url":"https://github.com/uber/horovod/issues/60#issuecomment-352102217","issue_url":"https://api.github.com/repos/uber/horovod/issues/60","id":352102217,"user":{"login":"atinsood","id":515920,"avatar_url":"https://avatars0.githubusercontent.com/u/515920?v=4","gravatar_id":"","url":"https://api.github.com/users/atinsood","html_url":"https://github.com/atinsood","followers_url":"https://api.github.com/users/atinsood/followers","following_url":"https://api.github.com/users/atinsood/following{/other_user}","gists_url":"https://api.github.com/users/atinsood/gists{/gist_id}","starred_url":"https://api.github.com/users/atinsood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/atinsood/subscriptions","organizations_url":"https://api.github.com/users/atinsood/orgs","repos_url":"https://api.github.com/users/atinsood/repos","events_url":"https://api.github.com/users/atinsood/events{/privacy}","received_events_url":"https://api.github.com/users/atinsood/received_events","type":"User","site_admin":false},"created_at":"2017-12-15T20:20:23Z","updated_at":"2017-12-15T20:20:23Z","author_association":"NONE","body":"yep yep, argh!!! that was it. switching to tensorflow-gpu  fixed things. thank you thank you, appreciate you taking the time and looking into it :)"}}
{"actor":{"display_login":"atinsood"}}