{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/5422","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/5422/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/5422/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/5422/events","html_url":"https://github.com/dmlc/mxnet/issues/5422","id":214602455,"number":5422,"title":"Very Slow MxNet distributed training ","user":{"login":"salemmohammed","id":6541193,"avatar_url":"https://avatars1.githubusercontent.com/u/6541193?v=3","gravatar_id":"","url":"https://api.github.com/users/salemmohammed","html_url":"https://github.com/salemmohammed","followers_url":"https://api.github.com/users/salemmohammed/followers","following_url":"https://api.github.com/users/salemmohammed/following{/other_user}","gists_url":"https://api.github.com/users/salemmohammed/gists{/gist_id}","starred_url":"https://api.github.com/users/salemmohammed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/salemmohammed/subscriptions","organizations_url":"https://api.github.com/users/salemmohammed/orgs","repos_url":"https://api.github.com/users/salemmohammed/repos","events_url":"https://api.github.com/users/salemmohammed/events{/privacy}","received_events_url":"https://api.github.com/users/salemmohammed/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-03-16T05:27:05Z","updated_at":"2017-03-16T18:56:14Z","closed_at":"2017-03-16T18:00:46Z","body":"I am working on the example training_mnist.py and I notice that it's very slow when I run the distributed training. Also, cross entropy value gave me Nan. When I ran it on my single machine, everything works fine but I believe it has slow speed training per sample.\r\n\r\nHere is the result for training on single machine:\r\n\r\n`python train_mnist.py --network mlp`\r\n\r\nINFO:root:Epoch[0] Batch [100]\tSpeed: 40726.16 samples/sec\tTrain-cross-entropy=1.054846\r\nINFO:root:Epoch[0] Batch [200]\tSpeed: 41420.49 samples/sec\tTrain-cross-entropy=0.544122\r\nINFO:root:Epoch[0] Batch [300]\tSpeed: 41170.83 samples/sec\tTrain-cross-entropy=0.471626\r\nINFO:root:Epoch[0] Batch [400]\tSpeed: 39867.68 samples/sec\tTrain-cross-entropy=0.437326\r\nINFO:root:Epoch[0] Batch [500]\tSpeed: 40247.01 samples/sec\tTrain-cross-entropy=0.414339\r\nINFO:root:Epoch[0] Batch [600]\tSpeed: 44374.19 samples/sec\tTrain-cross-entropy=0.396647\r\nINFO:root:Epoch[0] Batch [700]\tSpeed: 40458.16 samples/sec\tTrain-cross-entropy=0.377359\r\nINFO:root:Epoch[0] Batch [800]\tSpeed: 39988.00 samples/sec\tTrain-cross-entropy=0.360503\r\nINFO:root:Epoch[0] Batch [900]\tSpeed: 40203.55 samples/sec\tTrain-cross-entropy=0.358936\r\nINFO:root:Epoch[0] Train-cross-entropy=0.350466\r\nINFO:root:Epoch[0] Time cost=1.476\r\n\r\nI changed map.py file to:\r\n`\r\ndef get_symbol(num_classes=10, **kwargs):\r\n    data = mx.symbol.Variable('data')\r\n    data = mx.sym.Flatten(data=data)\r\n    fc3  = mx.symbol.FullyConnected(data = data, name='fc3', num_hidden=num_classes)\r\n    mlp  = mx.symbol.SoftmaxOutput(data = fc3, name = 'softmax')\r\n    return mlp\r\n`\r\n\r\nHere is the result for distributed training.\r\n`../../tools/launch.py -n 3 -H hosts --sync-dst-dir /tmp/mxnet python train_mnist.py --network mlp --kv-store dist_sync`\r\n\r\nINFO:root:Epoch[0] Batch [100]\tSpeed: 149.58 samples/sec\tTrain-cross-entropy=nan\r\nINFO:root:Epoch[0] Batch [100]\tSpeed: 149.42 samples/sec\tTrain-cross-entropy=nan\r\nINFO:root:Epoch[0] Batch [100]\tSpeed: 149.78 samples/sec\tTrain-cross-entropy=nan\r\n\r\n\r\nFinally, Does TensorFlow faster than Mxnet ?\r\n\r\n\r\nSincerely\r\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/287157560","html_url":"https://github.com/dmlc/mxnet/issues/5422#issuecomment-287157560","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/5422","id":287157560,"user":{"login":"salemmohammed","id":6541193,"avatar_url":"https://avatars1.githubusercontent.com/u/6541193?v=3","gravatar_id":"","url":"https://api.github.com/users/salemmohammed","html_url":"https://github.com/salemmohammed","followers_url":"https://api.github.com/users/salemmohammed/followers","following_url":"https://api.github.com/users/salemmohammed/following{/other_user}","gists_url":"https://api.github.com/users/salemmohammed/gists{/gist_id}","starred_url":"https://api.github.com/users/salemmohammed/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/salemmohammed/subscriptions","organizations_url":"https://api.github.com/users/salemmohammed/orgs","repos_url":"https://api.github.com/users/salemmohammed/repos","events_url":"https://api.github.com/users/salemmohammed/events{/privacy}","received_events_url":"https://api.github.com/users/salemmohammed/received_events","type":"User","site_admin":false},"created_at":"2017-03-16T18:56:14Z","updated_at":"2017-03-16T18:56:14Z","body":"@philipskokoh \r\nI got why the distributed training is slow but I still wondering why the cross entropy is Nan.\r\nMay you please give me a hint ?"}}
{"actor":{"display_login":"salemmohammed"}}