{"action":"created","comment":{"url":"https://api.github.com/repos/awslabs/sockeye/pulls/comments/154727911","pull_request_review_id":80933447,"id":154727911,"diff_hunk":"@@ -1021,7 +1060,8 @@ def decode_step(self,\n         new_layer_states = []\n \n         # (batch_size, num_embed)\n-        target_embed_prev = self.pos_embedding.encode_positions(indices, target_embed_prev)\n+        symbolic_step = mx.sym.arange(start=step - 1, stop=step, step=1, name='symbolic_step')","path":"sockeye/decoder.py","position":396,"original_position":396,"commit_id":"78a4333f40cbe9ac436f4606a2691a9d2a8f85bc","original_commit_id":"78a4333f40cbe9ac436f4606a2691a9d2a8f85bc","user":{"login":"tdomhan","id":363737,"avatar_url":"https://avatars1.githubusercontent.com/u/363737?v=4","gravatar_id":"","url":"https://api.github.com/users/tdomhan","html_url":"https://github.com/tdomhan","followers_url":"https://api.github.com/users/tdomhan/followers","following_url":"https://api.github.com/users/tdomhan/following{/other_user}","gists_url":"https://api.github.com/users/tdomhan/gists{/gist_id}","starred_url":"https://api.github.com/users/tdomhan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tdomhan/subscriptions","organizations_url":"https://api.github.com/users/tdomhan/orgs","repos_url":"https://api.github.com/users/tdomhan/repos","events_url":"https://api.github.com/users/tdomhan/events{/privacy}","received_events_url":"https://api.github.com/users/tdomhan/received_events","type":"User","site_admin":false},"body":"minor:\r\nso symbolic step is actually step minus one, no? Would it make sense to call the result `indices` as before?\r\n```\r\nindices = mx.sym.arange(start=step - 1, stop=step, step=1, name='indices')\r\n```","created_at":"2017-12-04T18:07:36Z","updated_at":"2017-12-04T18:16:47Z","html_url":"https://github.com/awslabs/sockeye/pull/226#discussion_r154727911","pull_request_url":"https://api.github.com/repos/awslabs/sockeye/pulls/226","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/awslabs/sockeye/pulls/comments/154727911"},"html":{"href":"https://github.com/awslabs/sockeye/pull/226#discussion_r154727911"},"pull_request":{"href":"https://api.github.com/repos/awslabs/sockeye/pulls/226"}}},"pull_request":{"url":"https://api.github.com/repos/awslabs/sockeye/pulls/226","id":155154595,"html_url":"https://github.com/awslabs/sockeye/pull/226","diff_url":"https://github.com/awslabs/sockeye/pull/226.diff","patch_url":"https://github.com/awslabs/sockeye/pull/226.patch","issue_url":"https://api.github.com/repos/awslabs/sockeye/issues/226","number":226,"state":"open","locked":false,"title":"Fast decoding for transformer models. Caches keys and values of self-attention before softmax.","user":{"login":"fhieber","id":2027990,"avatar_url":"https://avatars2.githubusercontent.com/u/2027990?v=4","gravatar_id":"","url":"https://api.github.com/users/fhieber","html_url":"https://github.com/fhieber","followers_url":"https://api.github.com/users/fhieber/followers","following_url":"https://api.github.com/users/fhieber/following{/other_user}","gists_url":"https://api.github.com/users/fhieber/gists{/gist_id}","starred_url":"https://api.github.com/users/fhieber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fhieber/subscriptions","organizations_url":"https://api.github.com/users/fhieber/orgs","repos_url":"https://api.github.com/users/fhieber/repos","events_url":"https://api.github.com/users/fhieber/events{/privacy}","received_events_url":"https://api.github.com/users/fhieber/received_events","type":"User","site_admin":false},"body":"This adds fast decoding for transformers by caching intermediate, previous outputs.\r\nFor this to work, inference is now using buckets of size 1 on the target side by default.\r\nAs a side effect, we can also again feed in only the previous word instead of the complete target sequences to decoder modules.\r\nI also changed the implemention so that we create bias tensors for masking attentions only once, and not for every layer using SequenceMask.\r\n\r\nThe cache tensors are also fed into and out of the decoder module as a single, concatenated tensor which turned out to be more efficient.\r\n\r\nAlso changed finished hyps logic in beam search a little bit so that finished hypotheses can never become 'unfinished'.\r\n\r\n#### Pull Request Checklist ##\r\n- [x] Changes are complete (if posting work-in-progress code, prefix your pull request title with '[WIP]'\r\nuntil you can check this box.\r\n- [x] Unit tests pass (`pytest`)\r\n- [x] System tests pass (`pytest test/system`)\r\n- [x] Passed code style checking (`./pre-commit.sh` or manual run of pylint & mypy)\r\n- [x] You have considered writing a test\r\n- [x] Updated major/minor version in `sockeye/__init__.py`. Major version bump if this is a backwards incompatible change.\r\n- [x] Updated CHANGELOG.md\r\n\r\n","created_at":"2017-11-28T18:55:17Z","updated_at":"2017-12-04T18:16:47Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e104bf1c622796913731b8df8194f1386b5db5d2","assignee":null,"assignees":[],"requested_reviewers":[{"login":"davvil","id":683589,"avatar_url":"https://avatars0.githubusercontent.com/u/683589?v=4","gravatar_id":"","url":"https://api.github.com/users/davvil","html_url":"https://github.com/davvil","followers_url":"https://api.github.com/users/davvil/followers","following_url":"https://api.github.com/users/davvil/following{/other_user}","gists_url":"https://api.github.com/users/davvil/gists{/gist_id}","starred_url":"https://api.github.com/users/davvil/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davvil/subscriptions","organizations_url":"https://api.github.com/users/davvil/orgs","repos_url":"https://api.github.com/users/davvil/repos","events_url":"https://api.github.com/users/davvil/events{/privacy}","received_events_url":"https://api.github.com/users/davvil/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/awslabs/sockeye/pulls/226/commits","review_comments_url":"https://api.github.com/repos/awslabs/sockeye/pulls/226/comments","review_comment_url":"https://api.github.com/repos/awslabs/sockeye/pulls/comments{/number}","comments_url":"https://api.github.com/repos/awslabs/sockeye/issues/226/comments","statuses_url":"https://api.github.com/repos/awslabs/sockeye/statuses/78a4333f40cbe9ac436f4606a2691a9d2a8f85bc","head":{"label":"awslabs:fast_decode","ref":"fast_decode","sha":"78a4333f40cbe9ac436f4606a2691a9d2a8f85bc","user":{"login":"awslabs","id":3299148,"avatar_url":"https://avatars0.githubusercontent.com/u/3299148?v=4","gravatar_id":"","url":"https://api.github.com/users/awslabs","html_url":"https://github.com/awslabs","followers_url":"https://api.github.com/users/awslabs/followers","following_url":"https://api.github.com/users/awslabs/following{/other_user}","gists_url":"https://api.github.com/users/awslabs/gists{/gist_id}","starred_url":"https://api.github.com/users/awslabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awslabs/subscriptions","organizations_url":"https://api.github.com/users/awslabs/orgs","repos_url":"https://api.github.com/users/awslabs/repos","events_url":"https://api.github.com/users/awslabs/events{/privacy}","received_events_url":"https://api.github.com/users/awslabs/received_events","type":"Organization","site_admin":false},"repo":{"id":93721432,"name":"sockeye","full_name":"awslabs/sockeye","owner":{"login":"awslabs","id":3299148,"avatar_url":"https://avatars0.githubusercontent.com/u/3299148?v=4","gravatar_id":"","url":"https://api.github.com/users/awslabs","html_url":"https://github.com/awslabs","followers_url":"https://api.github.com/users/awslabs/followers","following_url":"https://api.github.com/users/awslabs/following{/other_user}","gists_url":"https://api.github.com/users/awslabs/gists{/gist_id}","starred_url":"https://api.github.com/users/awslabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awslabs/subscriptions","organizations_url":"https://api.github.com/users/awslabs/orgs","repos_url":"https://api.github.com/users/awslabs/repos","events_url":"https://api.github.com/users/awslabs/events{/privacy}","received_events_url":"https://api.github.com/users/awslabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/awslabs/sockeye","description":"Sequence-to-sequence framework with a focus on Neural Machine Translation based on Apache MXNet","fork":false,"url":"https://api.github.com/repos/awslabs/sockeye","forks_url":"https://api.github.com/repos/awslabs/sockeye/forks","keys_url":"https://api.github.com/repos/awslabs/sockeye/keys{/key_id}","collaborators_url":"https://api.github.com/repos/awslabs/sockeye/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/awslabs/sockeye/teams","hooks_url":"https://api.github.com/repos/awslabs/sockeye/hooks","issue_events_url":"https://api.github.com/repos/awslabs/sockeye/issues/events{/number}","events_url":"https://api.github.com/repos/awslabs/sockeye/events","assignees_url":"https://api.github.com/repos/awslabs/sockeye/assignees{/user}","branches_url":"https://api.github.com/repos/awslabs/sockeye/branches{/branch}","tags_url":"https://api.github.com/repos/awslabs/sockeye/tags","blobs_url":"https://api.github.com/repos/awslabs/sockeye/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/awslabs/sockeye/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/awslabs/sockeye/git/refs{/sha}","trees_url":"https://api.github.com/repos/awslabs/sockeye/git/trees{/sha}","statuses_url":"https://api.github.com/repos/awslabs/sockeye/statuses/{sha}","languages_url":"https://api.github.com/repos/awslabs/sockeye/languages","stargazers_url":"https://api.github.com/repos/awslabs/sockeye/stargazers","contributors_url":"https://api.github.com/repos/awslabs/sockeye/contributors","subscribers_url":"https://api.github.com/repos/awslabs/sockeye/subscribers","subscription_url":"https://api.github.com/repos/awslabs/sockeye/subscription","commits_url":"https://api.github.com/repos/awslabs/sockeye/commits{/sha}","git_commits_url":"https://api.github.com/repos/awslabs/sockeye/git/commits{/sha}","comments_url":"https://api.github.com/repos/awslabs/sockeye/comments{/number}","issue_comment_url":"https://api.github.com/repos/awslabs/sockeye/issues/comments{/number}","contents_url":"https://api.github.com/repos/awslabs/sockeye/contents/{+path}","compare_url":"https://api.github.com/repos/awslabs/sockeye/compare/{base}...{head}","merges_url":"https://api.github.com/repos/awslabs/sockeye/merges","archive_url":"https://api.github.com/repos/awslabs/sockeye/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/awslabs/sockeye/downloads","issues_url":"https://api.github.com/repos/awslabs/sockeye/issues{/number}","pulls_url":"https://api.github.com/repos/awslabs/sockeye/pulls{/number}","milestones_url":"https://api.github.com/repos/awslabs/sockeye/milestones{/number}","notifications_url":"https://api.github.com/repos/awslabs/sockeye/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/awslabs/sockeye/labels{/name}","releases_url":"https://api.github.com/repos/awslabs/sockeye/releases{/id}","deployments_url":"https://api.github.com/repos/awslabs/sockeye/deployments","created_at":"2017-06-08T07:44:30Z","updated_at":"2017-12-04T16:44:15Z","pushed_at":"2017-12-04T16:44:01Z","git_url":"git://github.com/awslabs/sockeye.git","ssh_url":"git@github.com:awslabs/sockeye.git","clone_url":"https://github.com/awslabs/sockeye.git","svn_url":"https://github.com/awslabs/sockeye","homepage":"","size":1420,"stargazers_count":282,"watchers_count":282,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":78,"mirror_url":null,"archived":false,"open_issues_count":12,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":78,"open_issues":12,"watchers":282,"default_branch":"master"}},"base":{"label":"awslabs:master","ref":"master","sha":"afc884d6fe0bef97baaad32dc52e91a8b7951e11","user":{"login":"awslabs","id":3299148,"avatar_url":"https://avatars0.githubusercontent.com/u/3299148?v=4","gravatar_id":"","url":"https://api.github.com/users/awslabs","html_url":"https://github.com/awslabs","followers_url":"https://api.github.com/users/awslabs/followers","following_url":"https://api.github.com/users/awslabs/following{/other_user}","gists_url":"https://api.github.com/users/awslabs/gists{/gist_id}","starred_url":"https://api.github.com/users/awslabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awslabs/subscriptions","organizations_url":"https://api.github.com/users/awslabs/orgs","repos_url":"https://api.github.com/users/awslabs/repos","events_url":"https://api.github.com/users/awslabs/events{/privacy}","received_events_url":"https://api.github.com/users/awslabs/received_events","type":"Organization","site_admin":false},"repo":{"id":93721432,"name":"sockeye","full_name":"awslabs/sockeye","owner":{"login":"awslabs","id":3299148,"avatar_url":"https://avatars0.githubusercontent.com/u/3299148?v=4","gravatar_id":"","url":"https://api.github.com/users/awslabs","html_url":"https://github.com/awslabs","followers_url":"https://api.github.com/users/awslabs/followers","following_url":"https://api.github.com/users/awslabs/following{/other_user}","gists_url":"https://api.github.com/users/awslabs/gists{/gist_id}","starred_url":"https://api.github.com/users/awslabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awslabs/subscriptions","organizations_url":"https://api.github.com/users/awslabs/orgs","repos_url":"https://api.github.com/users/awslabs/repos","events_url":"https://api.github.com/users/awslabs/events{/privacy}","received_events_url":"https://api.github.com/users/awslabs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/awslabs/sockeye","description":"Sequence-to-sequence framework with a focus on Neural Machine Translation based on Apache MXNet","fork":false,"url":"https://api.github.com/repos/awslabs/sockeye","forks_url":"https://api.github.com/repos/awslabs/sockeye/forks","keys_url":"https://api.github.com/repos/awslabs/sockeye/keys{/key_id}","collaborators_url":"https://api.github.com/repos/awslabs/sockeye/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/awslabs/sockeye/teams","hooks_url":"https://api.github.com/repos/awslabs/sockeye/hooks","issue_events_url":"https://api.github.com/repos/awslabs/sockeye/issues/events{/number}","events_url":"https://api.github.com/repos/awslabs/sockeye/events","assignees_url":"https://api.github.com/repos/awslabs/sockeye/assignees{/user}","branches_url":"https://api.github.com/repos/awslabs/sockeye/branches{/branch}","tags_url":"https://api.github.com/repos/awslabs/sockeye/tags","blobs_url":"https://api.github.com/repos/awslabs/sockeye/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/awslabs/sockeye/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/awslabs/sockeye/git/refs{/sha}","trees_url":"https://api.github.com/repos/awslabs/sockeye/git/trees{/sha}","statuses_url":"https://api.github.com/repos/awslabs/sockeye/statuses/{sha}","languages_url":"https://api.github.com/repos/awslabs/sockeye/languages","stargazers_url":"https://api.github.com/repos/awslabs/sockeye/stargazers","contributors_url":"https://api.github.com/repos/awslabs/sockeye/contributors","subscribers_url":"https://api.github.com/repos/awslabs/sockeye/subscribers","subscription_url":"https://api.github.com/repos/awslabs/sockeye/subscription","commits_url":"https://api.github.com/repos/awslabs/sockeye/commits{/sha}","git_commits_url":"https://api.github.com/repos/awslabs/sockeye/git/commits{/sha}","comments_url":"https://api.github.com/repos/awslabs/sockeye/comments{/number}","issue_comment_url":"https://api.github.com/repos/awslabs/sockeye/issues/comments{/number}","contents_url":"https://api.github.com/repos/awslabs/sockeye/contents/{+path}","compare_url":"https://api.github.com/repos/awslabs/sockeye/compare/{base}...{head}","merges_url":"https://api.github.com/repos/awslabs/sockeye/merges","archive_url":"https://api.github.com/repos/awslabs/sockeye/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/awslabs/sockeye/downloads","issues_url":"https://api.github.com/repos/awslabs/sockeye/issues{/number}","pulls_url":"https://api.github.com/repos/awslabs/sockeye/pulls{/number}","milestones_url":"https://api.github.com/repos/awslabs/sockeye/milestones{/number}","notifications_url":"https://api.github.com/repos/awslabs/sockeye/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/awslabs/sockeye/labels{/name}","releases_url":"https://api.github.com/repos/awslabs/sockeye/releases{/id}","deployments_url":"https://api.github.com/repos/awslabs/sockeye/deployments","created_at":"2017-06-08T07:44:30Z","updated_at":"2017-12-04T16:44:15Z","pushed_at":"2017-12-04T16:44:01Z","git_url":"git://github.com/awslabs/sockeye.git","ssh_url":"git@github.com:awslabs/sockeye.git","clone_url":"https://github.com/awslabs/sockeye.git","svn_url":"https://github.com/awslabs/sockeye","homepage":"","size":1420,"stargazers_count":282,"watchers_count":282,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":78,"mirror_url":null,"archived":false,"open_issues_count":12,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0"},"forks":78,"open_issues":12,"watchers":282,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/awslabs/sockeye/pulls/226"},"html":{"href":"https://github.com/awslabs/sockeye/pull/226"},"issue":{"href":"https://api.github.com/repos/awslabs/sockeye/issues/226"},"comments":{"href":"https://api.github.com/repos/awslabs/sockeye/issues/226/comments"},"review_comments":{"href":"https://api.github.com/repos/awslabs/sockeye/pulls/226/comments"},"review_comment":{"href":"https://api.github.com/repos/awslabs/sockeye/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/awslabs/sockeye/pulls/226/commits"},"statuses":{"href":"https://api.github.com/repos/awslabs/sockeye/statuses/78a4333f40cbe9ac436f4606a2691a9d2a8f85bc"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"tdomhan"}}