{"action":"closed","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/3327","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/3327/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/3327/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/3327/events","html_url":"https://github.com/fchollet/keras/issues/3327","id":167787499,"number":3327,"title":"Lack of Parallelism for a relatively simple model","user":{"login":"sinjax","id":52549,"avatar_url":"https://avatars3.githubusercontent.com/u/52549?v=3","gravatar_id":"","url":"https://api.github.com/users/sinjax","html_url":"https://github.com/sinjax","followers_url":"https://api.github.com/users/sinjax/followers","following_url":"https://api.github.com/users/sinjax/following{/other_user}","gists_url":"https://api.github.com/users/sinjax/gists{/gist_id}","starred_url":"https://api.github.com/users/sinjax/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sinjax/subscriptions","organizations_url":"https://api.github.com/users/sinjax/orgs","repos_url":"https://api.github.com/users/sinjax/repos","events_url":"https://api.github.com/users/sinjax/events{/privacy}","received_events_url":"https://api.github.com/users/sinjax/received_events","type":"User","site_admin":false},"labels":[{"id":611595073,"url":"https://api.github.com/repos/fchollet/keras/labels/stale","name":"stale","color":"ffffff","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-27T07:19:12Z","updated_at":"2017-06-22T22:14:01Z","closed_at":"2017-06-22T22:14:01Z","body":"Hey All,\n\nFirst off thanks for all the amazing work on keras. \nI've been using it on and off for a year now and I'm really enjoying the way keras makes me think about the problems I've been modeling. \n\nRight, so I have a keras install on a 32 core machine. When running a seq2seq like this:\n\n``` python\nself.model = Sequential()\nself.model.add(Embedding(len(self.vocab), embedding_size,mask_zero=True))\nself.model.add(LSTM(hidden_size)) \nself.model.add(Dense(hidden_size))\nself.model.add(Activation('relu'))\nself.model.add(RepeatVector(max_sentence_length))\nself.model.add(LSTM( hidden_size, return_sequences=True))\nself.model.add(TimeDistributed(Dense(len(self.vocab))))\nself.model.add(Activation('softmax'))\nself.model.compile(loss='categorical_crossentropy', optimizer='adam')\n```\n\nI do a top and I see CPU usage to be around 3200% and average load hovers around 18 to 20. \nSo this seems to parallelize.\n\nHowever, when running a relatively simple model like this:\n\n``` python\ndef my_merge(parts):\n  a, b, c = parts\n  d = K.sqrt(K.sum(K.square(a + b - c), axis=2))\n  return d\na = Input(shape=(1,), dtype='int32')\nb = Input(shape=(1,), dtype='int32')\nc = Input(shape=(1,), dtype='int32')\nself.emb1 = Embedding(emb1_size, embedding_dims, W_constraint=unitnorm())\nself.emb2 = Embedding(emb2_size, embedding_dims)\n\na_embed = self.emb1(a)\nb_embed = self.emb2(b)\nc_embed = self.emb1(c)\n\nfinal_layer = merge([a_embed, b_embed, c_embed], mode=my_merge, output_shape=(1,))\nmodel = Model(input=[left, relation, right], output=final_layer, name=name)\n```\n\nI do a top and I see CPU usage is around 100% and average load hovers around 1 or 1.08.\nSo this does not seem to parallelize\n\nIn terms of their input both models are fed via a generator.\n\nAm i wrong to expect parallelism here? My assumption was an operation level parallelism \nand seeing as all I'm doing here is summation it feels like that should parallise efficiently, \nis this not correct?\n\nTo be fair, this relatively simple model seems to scale fairly well to huge batch sizes.\nStill, I wonder if i'm missing a trick here.\nAlso I'm experimenting with more complex variations of this simple model, things that include\na couple dot products in the merge applied on outputs of another embedding layer, and still I don't seem to get any multi-core usage\n\nIf it helps, emb1_size can go as high as 30 million and emb2 size around 1000 or so.\nI'm using the theano backend. \nThe seq2seq model is being trained on batch sizes of 1024 while with the embedding model batches of 1024 seem to take the same time as batches of 100k! \n\nThanks!\n"}}
{"actor":{"display_login":"probot-stale"}}