{"action":"created","issue":{"url":"https://api.github.com/repos/bdhingra/ga-reader/issues/1","repository_url":"https://api.github.com/repos/bdhingra/ga-reader","labels_url":"https://api.github.com/repos/bdhingra/ga-reader/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/bdhingra/ga-reader/issues/1/comments","events_url":"https://api.github.com/repos/bdhingra/ga-reader/issues/1/events","html_url":"https://github.com/bdhingra/ga-reader/issues/1","id":221128459,"number":1,"title":"It takes so long to use GPU for training: still 0 epoch after a day.","user":{"login":"DaehanKim","id":20675681,"avatar_url":"https://avatars1.githubusercontent.com/u/20675681?v=3","gravatar_id":"","url":"https://api.github.com/users/DaehanKim","html_url":"https://github.com/DaehanKim","followers_url":"https://api.github.com/users/DaehanKim/followers","following_url":"https://api.github.com/users/DaehanKim/following{/other_user}","gists_url":"https://api.github.com/users/DaehanKim/gists{/gist_id}","starred_url":"https://api.github.com/users/DaehanKim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DaehanKim/subscriptions","organizations_url":"https://api.github.com/users/DaehanKim/orgs","repos_url":"https://api.github.com/users/DaehanKim/repos","events_url":"https://api.github.com/users/DaehanKim/events{/privacy}","received_events_url":"https://api.github.com/users/DaehanKim/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-04-12T01:49:57Z","updated_at":"2017-04-13T16:29:13Z","closed_at":null,"body":"```\r\n->  python run.py\r\nUsing gpu device 0: TITAN X (Pascal) (CNMeM is disabled, cuDNN 5105)\r\n/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/__init__.py:600: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.\r\n  warnings.warn(warn)\r\nloading vocabularies from data/wdw/vocab.txt ...\r\nvocab_size = 324613\r\nnum characters = 102\r\n0 anonymoused entities\r\n324613 other tokens (including @placeholder, @begin and @end)\r\npreparing training data ...\r\n@placeholder not found in  data/wdw/training/AFP_ENG_20040114.0037.question . Fixing...\r\n@placeholder not found in  data/wdw/training/CNA_ENG_19990325.0027.question . Fixing...\r\n@placeholder not found in  data/wdw/training/AFP_ENG_20020701.0566.question . Fixing...\r\npreparing validation data ...\r\npreparing test data ...\r\nbuilding minibatch loaders ...\r\nbuilding network ...\r\n167991/324613 vocabs are initialized with word2vec embeddings.\r\nWARNING (theano.tensor.blas): We did not found a dynamic library into the library_dir of the library we use for blas. If you use ATLAS, make sure to compile it with dynamics library.\r\ntraining ...\r\nloading previously saved model\r\n...\r\nEpoch 0 TRAIN loss=8.4052e-01 acc=0.7188 elapsed=75252.5\r\nEpoch 0 TRAIN loss=4.3968e-01 acc=0.8438 elapsed=75258.4\r\nEpoch 0 TRAIN loss=1.0990e+00 acc=0.5312 elapsed=75271.3\r\nEpoch 0 TRAIN loss=7.4241e-01 acc=0.5625 elapsed=75279.6\r\n```\r\n\r\nIt uses only a portion of TITAN X, such as 300~400MB, while it has two TITAN X of 12GB each.\r\nI tried `libgpuarray` but it is too complicated to follow up. I actually did everything its instruction says but can't figure out how I can apply `libgpuarray` to individual projects I need. Is there any `.rc` file needed? Any help will be much appreciated! Thank you in advance."},"comment":{"url":"https://api.github.com/repos/bdhingra/ga-reader/issues/comments/293950388","html_url":"https://github.com/bdhingra/ga-reader/issues/1#issuecomment-293950388","issue_url":"https://api.github.com/repos/bdhingra/ga-reader/issues/1","id":293950388,"user":{"login":"bdhingra","id":8645980,"avatar_url":"https://avatars0.githubusercontent.com/u/8645980?v=3","gravatar_id":"","url":"https://api.github.com/users/bdhingra","html_url":"https://github.com/bdhingra","followers_url":"https://api.github.com/users/bdhingra/followers","following_url":"https://api.github.com/users/bdhingra/following{/other_user}","gists_url":"https://api.github.com/users/bdhingra/gists{/gist_id}","starred_url":"https://api.github.com/users/bdhingra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdhingra/subscriptions","organizations_url":"https://api.github.com/users/bdhingra/orgs","repos_url":"https://api.github.com/users/bdhingra/repos","events_url":"https://api.github.com/users/bdhingra/events{/privacy}","received_events_url":"https://api.github.com/users/bdhingra/received_events","type":"User","site_admin":false},"created_at":"2017-04-13T16:29:13Z","updated_at":"2017-04-13T16:29:13Z","body":"Can you try building a new model from scratch? Right now it is loading a previously saved model from the `experiments/` directory. \r\n\r\nTo make it build a new model, delete or move the existing models from the `experiments/` directory. \r\n\r\nAlso make sure that you run with the following flag: `floatX=float32`. You can either add this to `~/.theanorc` file, or prefix `THEANO_FLAGS=floatX=float32` to the python command you run."}}
{"actor":{"display_login":"bdhingra"}}