{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/621","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/621/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/621/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/621/events","html_url":"https://github.com/tensorflow/models/issues/621","id":187942460,"number":621,"title":"TextSum does not actually check the presence of Tags in vocabulary","user":{"login":"pranay360","id":8237595,"avatar_url":"https://avatars.githubusercontent.com/u/8237595?v=3","gravatar_id":"","url":"https://api.github.com/users/pranay360","html_url":"https://github.com/pranay360","followers_url":"https://api.github.com/users/pranay360/followers","following_url":"https://api.github.com/users/pranay360/following{/other_user}","gists_url":"https://api.github.com/users/pranay360/gists{/gist_id}","starred_url":"https://api.github.com/users/pranay360/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pranay360/subscriptions","organizations_url":"https://api.github.com/users/pranay360/orgs","repos_url":"https://api.github.com/users/pranay360/repos","events_url":"https://api.github.com/users/pranay360/events{/privacy}","received_events_url":"https://api.github.com/users/pranay360/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"peterjliu","id":5204812,"avatar_url":"https://avatars.githubusercontent.com/u/5204812?v=3","gravatar_id":"","url":"https://api.github.com/users/peterjliu","html_url":"https://github.com/peterjliu","followers_url":"https://api.github.com/users/peterjliu/followers","following_url":"https://api.github.com/users/peterjliu/following{/other_user}","gists_url":"https://api.github.com/users/peterjliu/gists{/gist_id}","starred_url":"https://api.github.com/users/peterjliu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterjliu/subscriptions","organizations_url":"https://api.github.com/users/peterjliu/orgs","repos_url":"https://api.github.com/users/peterjliu/repos","events_url":"https://api.github.com/users/peterjliu/events{/privacy}","received_events_url":"https://api.github.com/users/peterjliu/received_events","type":"User","site_admin":false},"assignees":[{"login":"peterjliu","id":5204812,"avatar_url":"https://avatars.githubusercontent.com/u/5204812?v=3","gravatar_id":"","url":"https://api.github.com/users/peterjliu","html_url":"https://github.com/peterjliu","followers_url":"https://api.github.com/users/peterjliu/followers","following_url":"https://api.github.com/users/peterjliu/following{/other_user}","gists_url":"https://api.github.com/users/peterjliu/gists{/gist_id}","starred_url":"https://api.github.com/users/peterjliu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterjliu/subscriptions","organizations_url":"https://api.github.com/users/peterjliu/orgs","repos_url":"https://api.github.com/users/peterjliu/repos","events_url":"https://api.github.com/users/peterjliu/events{/privacy}","received_events_url":"https://api.github.com/users/peterjliu/received_events","type":"User","site_admin":false}],"milestone":null,"comments":7,"created_at":"2016-11-08T09:34:13Z","updated_at":"2017-02-02T16:14:10Z","closed_at":"2016-12-08T17:49:51Z","body":"https://github.com/tensorflow/models/blob/master/textsum/seq2seq_attention.py#L163\r\n\r\nThe way of checking presence of required tokens is wrong because _**WordToId**_ function returns ID of Unknown <UNK> token **(which might be >=0 for most of the cases)** if the that particular token is not found, hence the scripts validates a wrong vocabulary file.\r\n\r\nI am training the TextSum algorithm on CNN Dataset and would also like to know about the presence of <PAD> in sample Toy Vocabulary, what should be the location/use of <PAD> in the data? Can someone provide a sample format for data including the <PAD> token, because I don't have it in my vocabulary.\r\nI would like to know how to include it in the data.\r\n\r\nPossible Fix: Raising an error if the WordToID function when the word is not found."},"comment":{"url":"https://api.github.com/repos/tensorflow/models/issues/comments/277002458","html_url":"https://github.com/tensorflow/models/issues/621#issuecomment-277002458","issue_url":"https://api.github.com/repos/tensorflow/models/issues/621","id":277002458,"user":{"login":"aanilpala","id":2991330,"avatar_url":"https://avatars.githubusercontent.com/u/2991330?v=3","gravatar_id":"","url":"https://api.github.com/users/aanilpala","html_url":"https://github.com/aanilpala","followers_url":"https://api.github.com/users/aanilpala/followers","following_url":"https://api.github.com/users/aanilpala/following{/other_user}","gists_url":"https://api.github.com/users/aanilpala/gists{/gist_id}","starred_url":"https://api.github.com/users/aanilpala/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aanilpala/subscriptions","organizations_url":"https://api.github.com/users/aanilpala/orgs","repos_url":"https://api.github.com/users/aanilpala/repos","events_url":"https://api.github.com/users/aanilpala/events{/privacy}","received_events_url":"https://api.github.com/users/aanilpala/received_events","type":"User","site_admin":false},"created_at":"2017-02-02T16:14:10Z","updated_at":"2017-02-02T16:14:10Z","body":"Hi,\r\n\r\nI couldn't find any more relevant bit of info anywhere about the pad token than here. I get that <PAD> should not appear in the data but in the vocabulary file. What I need to know based on what I should count them. Should I count one <PAD> for every shorter sentence than the maximum input size (which is enc_timesteps and set to 120) or should I count as many pad chars as the sentence falls short from the maximum input size character-wise? \r\n\r\nInterestingly, if I put a random frequency (bigger than 0) for <PAD> in my vocab file, things start working but then I get a memory error. I think the memory error is due to sth else though."}}
{"actor":{"display_login":"aanilpala"}}