{"action":"created","issue":{"url":"https://api.github.com/repos/csurfer/rake-nltk/issues/2","repository_url":"https://api.github.com/repos/csurfer/rake-nltk","labels_url":"https://api.github.com/repos/csurfer/rake-nltk/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/csurfer/rake-nltk/issues/2/comments","events_url":"https://api.github.com/repos/csurfer/rake-nltk/issues/2/events","html_url":"https://github.com/csurfer/rake-nltk/issues/2","id":203263496,"number":2,"title":"Weird results when running with text from Gutenberg corpus","user":{"login":"cgratie","id":9308958,"avatar_url":"https://avatars.githubusercontent.com/u/9308958?v=3","gravatar_id":"","url":"https://api.github.com/users/cgratie","html_url":"https://github.com/cgratie","followers_url":"https://api.github.com/users/cgratie/followers","following_url":"https://api.github.com/users/cgratie/following{/other_user}","gists_url":"https://api.github.com/users/cgratie/gists{/gist_id}","starred_url":"https://api.github.com/users/cgratie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cgratie/subscriptions","organizations_url":"https://api.github.com/users/cgratie/orgs","repos_url":"https://api.github.com/users/cgratie/repos","events_url":"https://api.github.com/users/cgratie/events{/privacy}","received_events_url":"https://api.github.com/users/cgratie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-01-26T00:32:15Z","updated_at":"2017-01-26T01:44:07Z","closed_at":null,"body":"I have tried this simple snippet to test keyword extraction on longer text using NLTK.\r\n```\r\nfrom nltk.corpus import gutenberg\r\nfrom rake_nltk.rake import Rake\r\nr = Rake()\r\nr.extract_keywords_from_text(gutenberg.raw(\"austen-emma.txt\"))\r\nfor phrase in r.get_ranked_phrases()[:5]:\r\n    print phrase\r\n```\r\nThe output I got is as follows:\r\n```\r\n.-- _she_ _felt_ _the_ _engagement_ _to_ _be_ _a_ _source_ _of_ _repentance_ _and_ _misery_ _to_ _each_\r\nbest time -- never tired -- every sort good -- hautboy infinitely superior --\r\nsaid nothing worth hearing -- looked without seeing -- admired without intelligence -- listened without knowing\r\nscarce -- chili preferred -- white wood finest flavour\r\nwithout scruple -- without apology -- without much apparent diffidence\r\n```\r\nI assume this is caused by either the word tokenizer or the sentence tokenizer, but I'm not sure how to fix it.\r\n"},"comment":{"url":"https://api.github.com/repos/csurfer/rake-nltk/issues/comments/275287666","html_url":"https://github.com/csurfer/rake-nltk/issues/2#issuecomment-275287666","issue_url":"https://api.github.com/repos/csurfer/rake-nltk/issues/2","id":275287666,"user":{"login":"csurfer","id":1235813,"avatar_url":"https://avatars.githubusercontent.com/u/1235813?v=3","gravatar_id":"","url":"https://api.github.com/users/csurfer","html_url":"https://github.com/csurfer","followers_url":"https://api.github.com/users/csurfer/followers","following_url":"https://api.github.com/users/csurfer/following{/other_user}","gists_url":"https://api.github.com/users/csurfer/gists{/gist_id}","starred_url":"https://api.github.com/users/csurfer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csurfer/subscriptions","organizations_url":"https://api.github.com/users/csurfer/orgs","repos_url":"https://api.github.com/users/csurfer/repos","events_url":"https://api.github.com/users/csurfer/events{/privacy}","received_events_url":"https://api.github.com/users/csurfer/received_events","type":"User","site_admin":false},"created_at":"2017-01-26T01:44:07Z","updated_at":"2017-01-26T01:44:07Z","body":"This behaviour is because the\r\n\r\n`nltk.tokenize.wordpunkt_tokenize()`\r\n\r\ndoesn't split punctuation clusters. i,e \r\n\r\n`nltk.tokenize.wordpunct_tokenize('--') => ['--']` and \r\n`nltk.tokenize.wordpunct_tokenize(';.') => [';.']`\r\n\r\nIt didn't seem like a good idea for the class to have cleaning and keyword extraction logic. So kept the cleaning logic out. Try cleaning the text before using the Keyword extraction.\r\n"}}
{"actor":{"display_login":"csurfer"}}