{"action":"created","issue":{"url":"https://api.github.com/repos/fchollet/keras/issues/4285","repository_url":"https://api.github.com/repos/fchollet/keras","labels_url":"https://api.github.com/repos/fchollet/keras/issues/4285/labels{/name}","comments_url":"https://api.github.com/repos/fchollet/keras/issues/4285/comments","events_url":"https://api.github.com/repos/fchollet/keras/issues/4285/events","html_url":"https://github.com/fchollet/keras/issues/4285","id":187231537,"number":4285,"title":"[Image captioning Vinyals style] Problem with Lambda (Merge) layer and batch size","user":{"login":"oarriaga","id":5577137,"avatar_url":"https://avatars3.githubusercontent.com/u/5577137?v=4","gravatar_id":"","url":"https://api.github.com/users/oarriaga","html_url":"https://github.com/oarriaga","followers_url":"https://api.github.com/users/oarriaga/followers","following_url":"https://api.github.com/users/oarriaga/following{/other_user}","gists_url":"https://api.github.com/users/oarriaga/gists{/gist_id}","starred_url":"https://api.github.com/users/oarriaga/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oarriaga/subscriptions","organizations_url":"https://api.github.com/users/oarriaga/orgs","repos_url":"https://api.github.com/users/oarriaga/repos","events_url":"https://api.github.com/users/oarriaga/events{/privacy}","received_events_url":"https://api.github.com/users/oarriaga/received_events","type":"User","site_admin":false},"labels":[{"id":611595073,"url":"https://api.github.com/repos/fchollet/keras/labels/stale","name":"stale","color":"ffffff","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-04T01:27:17Z","updated_at":"2017-08-01T15:57:35Z","closed_at":null,"body":"Hello everyone, \r\nI am currently trying to implement image captioning in Vinyals style. However, when I zero pad the features of the image; through a Lambda layer, so I can change the initial state of LSTM by summing it to the word embedding (Merge layer) I end up having the problem that batch seems to work only for a value of 1. \r\n\r\n```python\r\nEMBEDDING_DIM = 100\r\nMAX_CAPTION_LENGTH = 32\r\nVOCABULARY_SIZE = 10000 #len(words)\r\nbatch_nb = 1\r\n\r\ntensor_shape = Input(shape=(3,224, 224))\r\nbase_model = VGG19(weights='imagenet', include_top=False, input_tensor=tensor_shape)\r\nfor layer in base_model.layers:\r\n    layer.trainable = False\r\n\r\nimage_model = Sequential()\r\nimage_model.add(base_model)\r\nimage_model.add(Flatten())\r\nimage_model.add(Dense(EMBEDDING_DIM, activation='relu'))\r\n\r\ndef zero_pad_features(x):\r\n    y = K.zeros(shape=(MAX_CAPTION_LENGTH-1,EMBEDDING_DIM))\r\n    return K.concatenate((x,y),axis=0)\r\n\r\nimage_model.add(Lambda(zero_pad_features,output_shape=(MAX_CAPTION_LENGTH,EMBEDDING_DIM)))\r\n\r\nlanguage_model = Sequential()\r\nlanguage_model.add(Embedding(input_dim=VOCABULARY_SIZE, output_dim=EMBEDDING_DIM, input_length=MAX_CAPTION_LENGTH, mask_zero=True))\r\n\r\nmodel = Sequential()\r\nmodel.add(Merge([image_model,language_model],mode='sum'))\r\nmodel.add(LSTM(125, return_sequences=True))\r\nmodel.add(TimeDistributed(Dense(VOCABULARY_SIZE,activation='softmax')))\r\nmodel.compile(loss='categorical_crossentropy', optimizer='rmsprop',metrics=['accuracy'])\r\nprint('Starting training')\r\nmodel.fit([images, x_sequences], y, batch_size=batch_nb, nb_epoch=1, verbose=1)\r\n\r\nimg = np.ones((1,3,224,224))\r\ncaption = np.ones((1,32))\r\nprint(model.predict([img,caption]))\r\n```\r\n\r\nWhenever I change the value of **batch_nb** to something else it displays the following error \r\n```Python\r\n GpuElemwise. Input dimension mis-match. Input 1 (indices start at 0) has shape[1] == 32, but the output's size on that axis is 34.\r\nApply node that caused the error: GpuElemwise{add,no_inplace}(GpuDimShuffle{x,0,1}.0, GpuReshape{3}.0)\r\nToposort index: 438\r\nInputs types: [CudaNdarrayType(float32, (True, False, False)), CudaNdarrayType(float32, 3D)]\r\nInputs shapes: [(1, 34, 100), (3, 32, 100)]\r\nInputs strides: [(0, 100, 1), (3200, 100, 1)]\r\nInputs values: ['not shown', 'not shown']\r\nOutputs clients: [[GpuReshape{2}(GpuElemwise{add,no_inplace}.0, TensorConstant{[ -1 100]})]]\r\n```\r\n\r\nI believe that the error  **Inputs shapes: [(1, 34, 100), (3, 32, 100)]** is indicating that the input images are taking the second dimension as the batch dimension instead of taking the first dimension. \r\n\r\nThanks for taking the time,\r\nSincerely Octavio"},"comment":{"url":"https://api.github.com/repos/fchollet/keras/issues/comments/319414591","html_url":"https://github.com/fchollet/keras/issues/4285#issuecomment-319414591","issue_url":"https://api.github.com/repos/fchollet/keras/issues/4285","id":319414591,"user":{"login":"probot-stale[bot]","id":26384082,"avatar_url":"https://avatars2.githubusercontent.com/u/26350515?v=4","gravatar_id":"","url":"https://api.github.com/users/probot-stale%5Bbot%5D","html_url":"https://github.com/apps/probot-stale","followers_url":"https://api.github.com/users/probot-stale%5Bbot%5D/followers","following_url":"https://api.github.com/users/probot-stale%5Bbot%5D/following{/other_user}","gists_url":"https://api.github.com/users/probot-stale%5Bbot%5D/gists{/gist_id}","starred_url":"https://api.github.com/users/probot-stale%5Bbot%5D/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/probot-stale%5Bbot%5D/subscriptions","organizations_url":"https://api.github.com/users/probot-stale%5Bbot%5D/orgs","repos_url":"https://api.github.com/users/probot-stale%5Bbot%5D/repos","events_url":"https://api.github.com/users/probot-stale%5Bbot%5D/events{/privacy}","received_events_url":"https://api.github.com/users/probot-stale%5Bbot%5D/received_events","type":"Bot","site_admin":false},"created_at":"2017-08-01T15:57:35Z","updated_at":"2017-08-01T15:57:35Z","body":"This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity  occurs, but feel free to re-open a closed issue if needed.\n"}}
{"actor":{"display_login":"probot-stale"}}