{"action":"opened","issue":{"url":"https://api.github.com/repos/EderSantana/seya/issues/53","repository_url":"https://api.github.com/repos/EderSantana/seya","labels_url":"https://api.github.com/repos/EderSantana/seya/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/EderSantana/seya/issues/53/comments","events_url":"https://api.github.com/repos/EderSantana/seya/issues/53/events","html_url":"https://github.com/EderSantana/seya/issues/53","id":205461397,"number":53,"title":"How to insert Spatial Transformer layer BEFORE a convnet","user":{"login":"xavierdcruz0","id":13253500,"avatar_url":"https://avatars.githubusercontent.com/u/13253500?v=3","gravatar_id":"","url":"https://api.github.com/users/xavierdcruz0","html_url":"https://github.com/xavierdcruz0","followers_url":"https://api.github.com/users/xavierdcruz0/followers","following_url":"https://api.github.com/users/xavierdcruz0/following{/other_user}","gists_url":"https://api.github.com/users/xavierdcruz0/gists{/gist_id}","starred_url":"https://api.github.com/users/xavierdcruz0/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xavierdcruz0/subscriptions","organizations_url":"https://api.github.com/users/xavierdcruz0/orgs","repos_url":"https://api.github.com/users/xavierdcruz0/repos","events_url":"https://api.github.com/users/xavierdcruz0/events{/privacy}","received_events_url":"https://api.github.com/users/xavierdcruz0/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-02-05T22:50:47Z","updated_at":"2017-02-05T22:50:47Z","closed_at":null,"body":"I would like to have a Spatial Transformer layer before a pretrained convnet, such as the the Keras `ResNet50`. As such I have prepared the following various attempts to connect the `SpatialTransformer`'s output to the `ResNet50`'s input, but get errors every time. Furthermore, I'm not entirely sure how these layers are supposed to work with the Keras Functional API.\r\n\r\n```\r\nfrom seya.layers.attention import SpatialTransformer\r\nimport numpy as np\r\nfrom keras.layers import MaxPooling2D, Convolution2D, Dense, Activation, Flatten, Input, GlobalAveragePooling2D\r\nfrom keras.applications.resnet50 import ResNet50\r\nfrom keras.models import Model\r\nfrom keras.optimizers import Adam\r\n\r\n\r\ndef locnet(input):\r\n    # initial weights\r\n    b = np.zeros((2, 3), dtype='float32')\r\n    b[0, 0] = 1\r\n    b[1, 1] = 1\r\n    W = np.zeros((50, 6), dtype='float32')\r\n    weights = [W, b.flatten()]\r\n\r\n    # original from https://github.com/EderSantana/seya/blob/master/examples/Spatial%20Transformer%20Networks.ipynb\r\n    # locnet = Sequential()\r\n    # locnet.add(MaxPooling2D(pool_size=(2, 2), input_shape=input_shape))\r\n    # locnet.add(Convolution2D(20, 5, 5))\r\n    # locnet.add(MaxPooling2D(pool_size=(2, 2)))\r\n    # locnet.add(Convolution2D(20, 5, 5))\r\n    #\r\n    # locnet.add(Flatten())\r\n    # locnet.add(Dense(50))\r\n    # locnet.add(Activation('relu'))\r\n    # locnet.add(Dense(6, weights=weights))\r\n    # # locnet.add(Activation('sigmoid'))\r\n\r\n    # translated the above to functional API\r\n    pool1 = MaxPooling2D()(input)\r\n    conv1 = Convolution2D(20, 5, 5)(pool1)\r\n    pool2 = MaxPooling2D()(conv1)\r\n    conv2 = Convolution2D(20, 5, 5)(pool2)\r\n\r\n    flatten = Flatten()(conv2)\r\n    dense = Dense(50)(flatten)\r\n    # dense = Activation('relu')(dense)\r\n    params = Dense(6, weights=weights, name='affine_params')(dense)\r\n\r\n    return params\r\n\r\ndef spatial_transformer_net(input_shape, num_categories):\r\n    '''\r\n    plug a spatial transformer network into a Keras resnet50\r\n    '''\r\n\r\n    # make an input tensor\r\n    i = Input(input_shape)\r\n\r\n    # get a locnet\r\n    loc = locnet(i)\r\n\r\n    # get a spatial transformer\r\n    st = SpatialTransformer(localization_net=loc, downsample_factor=2, input_shape=input_shape)\r\n\r\n    # get a pretrained convnet\r\n    #######################################################can the SpatialTransformer be plugged in as the input tensor?\r\n    base_model = ResNet50(weights='imagenet', include_top=False)(st)\r\n    # freeze it\r\n    for layer in base_model.layers:\r\n        layer.trainable = False\r\n\r\n    ################################################################################## or do we plug it in here somehow?\r\n    # base_model.input = st.output\r\n    # base_model.input = st\r\n\r\n    # set output\r\n    Z = base_model.get_layer('activation_49').output\r\n    Z = GlobalAveragePooling2D()(Z)\r\n    Z = Dense(1024, activation='relu')(Z)\r\n    Z = Dense(num_categories, activation='softmax')(Z)\r\n\r\n    # create the Keras functional model\r\n    model = Model(input=i, output=Z)\r\n\r\n    # compile the model\r\n    model.compile(optimizer=Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0), loss='categorial_crossentropy', metrics=['accuracy'])\r\n\r\n    return model\r\n```"}}
{"actor":{"display_login":"xavierdcruz0"}}