{"action":"created","comment":{"url":"https://api.github.com/repos/explosion/spaCy/pulls/comments/145355766","pull_request_review_id":70141778,"id":145355766,"diff_hunk":"@@ -0,0 +1,177 @@\n+# coding: utf8\n+from __future__ import unicode_literals\n+\n+\n+# Source: https://github.com/taranjeet/hindi-tokenizer/blob/master/stopwords.txt\n+\n+STOP_WORDS = set(\"\"\"\n+अत\n+अपना\n+अपनी\n+अपने\n+अभी\n+अंदर\n+आदि\n+आप\n+इत्यादि\n+इन\n+इनका\n+इन्हीं\n+इन्हें\n+इन्हों\n+इस\n+इसका\n+इसकी\n+इसके\n+इसमें\n+इसी\n+इसे\n+उन\n+उनका\n+उनकी\n+उनके\n+उनको\n+उन्हीं\n+उन्हें\n+उन्हों\n+उस\n+उसके\n+उसी\n+उसे\n+एक\n+एवं\n+एस\n+ऐसे\n+और\n+कई\n+कर\n+करता\n+करते\n+करना\n+करने\n+करें\n+कहते\n+कहा\n+का","path":"spacy/lang/hi/stop_words.py","position":55,"original_position":55,"commit_id":"c0aceb9fbecfa0c62b3d3624b627a79e9984c040","original_commit_id":"c0aceb9fbecfa0c62b3d3624b627a79e9984c040","user":{"login":"ines","id":13643239,"avatar_url":"https://avatars0.githubusercontent.com/u/13643239?v=4","gravatar_id":"","url":"https://api.github.com/users/ines","html_url":"https://github.com/ines","followers_url":"https://api.github.com/users/ines/followers","following_url":"https://api.github.com/users/ines/following{/other_user}","gists_url":"https://api.github.com/users/ines/gists{/gist_id}","starred_url":"https://api.github.com/users/ines/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ines/subscriptions","organizations_url":"https://api.github.com/users/ines/orgs","repos_url":"https://api.github.com/users/ines/repos","events_url":"https://api.github.com/users/ines/events{/privacy}","received_events_url":"https://api.github.com/users/ines/received_events","type":"User","site_admin":false},"body":"Stop words don't have to be 100% perfect, but in general (also for tokenizer exceptions and other language data), we always try to make sure spaCy also covers the *actual* usage in the real world. For example, in the English stop words, including the word \"thru\" is almost more important and relevant than the word \"whence\" (which is usually pretty uncommon in the type of text people are using with spaCy). So keeping a good balance here is always good 👍","created_at":"2017-10-18T09:08:49Z","updated_at":"2017-10-18T09:08:49Z","html_url":"https://github.com/explosion/spaCy/pull/1425#discussion_r145355766","pull_request_url":"https://api.github.com/repos/explosion/spaCy/pulls/1425","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/explosion/spaCy/pulls/comments/145355766"},"html":{"href":"https://github.com/explosion/spaCy/pull/1425#discussion_r145355766"},"pull_request":{"href":"https://api.github.com/repos/explosion/spaCy/pulls/1425"}},"in_reply_to_id":145305516},"pull_request":{"url":"https://api.github.com/repos/explosion/spaCy/pulls/1425","id":146892619,"html_url":"https://github.com/explosion/spaCy/pull/1425","diff_url":"https://github.com/explosion/spaCy/pull/1425.diff","patch_url":"https://github.com/explosion/spaCy/pull/1425.patch","issue_url":"https://api.github.com/repos/explosion/spaCy/issues/1425","number":1425,"state":"open","locked":false,"title":"💫 Basic Hindi tokenization support","user":{"login":"ines","id":13643239,"avatar_url":"https://avatars0.githubusercontent.com/u/13643239?v=4","gravatar_id":"","url":"https://api.github.com/users/ines","html_url":"https://github.com/ines","followers_url":"https://api.github.com/users/ines/followers","following_url":"https://api.github.com/users/ines/following{/other_user}","gists_url":"https://api.github.com/users/ines/gists{/gist_id}","starred_url":"https://api.github.com/users/ines/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ines/subscriptions","organizations_url":"https://api.github.com/users/ines/orgs","repos_url":"https://api.github.com/users/ines/repos","events_url":"https://api.github.com/users/ines/events{/privacy}","received_events_url":"https://api.github.com/users/ines/received_events","type":"User","site_admin":false},"body":"Related issues: #641\r\nAlso CC'ing everyone who participated in the above issue:\r\n@kaustubhn, @gauravgr, @SanketSKasar, @souravsingh \r\n\r\nThis pull request adds a basic Hindi `Language` class to support tokenization with spaCy. It also includes a getter for the `NORM` attribute that adds the stem word if available (adapted from [here](http://computing.open.ac.uk/Sites/EACLSouthAsia/Papers/p6-Ramanathan.pdf)).\r\n\r\nSince Hindi support has been requested a lot in the past, I hope this will make it easier for others to contribute and improve the language data. (I don't know Hindi, so I was only able to add the very basics – feedback and contributions are appreciated!)\r\n\r\n## Todos and further ideas\r\n\r\n*  Add `LIKE_NUM` and `IS_DIGIT` getters and include the Hindi digit representations (see [here](https://github.com/explosion/spaCy/issues/641#issuecomment-336636719)). They should probably also be included in the `NORM`. One challenge here is to find the best and most performant way to check whether a string consists of Hindi digits.\r\n* Add tokenizer exceptions and [other language data](https://alpha.spacy.io/usage/adding-languages)\r\n* **Add tests!** Even a few simple examples covering text and expected tokens, or text and expected norms (stem words etc.) would be incredibly helpful.","created_at":"2017-10-16T21:25:47Z","updated_at":"2017-10-18T09:08:49Z","closed_at":null,"merged_at":null,"merge_commit_sha":"aca2b645346a95efc16a05140b85dfd809ae1141","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/explosion/spaCy/pulls/1425/commits","review_comments_url":"https://api.github.com/repos/explosion/spaCy/pulls/1425/comments","review_comment_url":"https://api.github.com/repos/explosion/spaCy/pulls/comments{/number}","comments_url":"https://api.github.com/repos/explosion/spaCy/issues/1425/comments","statuses_url":"https://api.github.com/repos/explosion/spaCy/statuses/c0aceb9fbecfa0c62b3d3624b627a79e9984c040","head":{"label":"explosion:feature/hindi-tokenizer","ref":"feature/hindi-tokenizer","sha":"c0aceb9fbecfa0c62b3d3624b627a79e9984c040","user":{"login":"explosion","id":20011530,"avatar_url":"https://avatars3.githubusercontent.com/u/20011530?v=4","gravatar_id":"","url":"https://api.github.com/users/explosion","html_url":"https://github.com/explosion","followers_url":"https://api.github.com/users/explosion/followers","following_url":"https://api.github.com/users/explosion/following{/other_user}","gists_url":"https://api.github.com/users/explosion/gists{/gist_id}","starred_url":"https://api.github.com/users/explosion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/explosion/subscriptions","organizations_url":"https://api.github.com/users/explosion/orgs","repos_url":"https://api.github.com/users/explosion/repos","events_url":"https://api.github.com/users/explosion/events{/privacy}","received_events_url":"https://api.github.com/users/explosion/received_events","type":"Organization","site_admin":false},"repo":{"id":21467110,"name":"spaCy","full_name":"explosion/spaCy","owner":{"login":"explosion","id":20011530,"avatar_url":"https://avatars3.githubusercontent.com/u/20011530?v=4","gravatar_id":"","url":"https://api.github.com/users/explosion","html_url":"https://github.com/explosion","followers_url":"https://api.github.com/users/explosion/followers","following_url":"https://api.github.com/users/explosion/following{/other_user}","gists_url":"https://api.github.com/users/explosion/gists{/gist_id}","starred_url":"https://api.github.com/users/explosion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/explosion/subscriptions","organizations_url":"https://api.github.com/users/explosion/orgs","repos_url":"https://api.github.com/users/explosion/repos","events_url":"https://api.github.com/users/explosion/events{/privacy}","received_events_url":"https://api.github.com/users/explosion/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/explosion/spaCy","description":"💫 Industrial-strength Natural Language Processing (NLP) with Python and Cython","fork":false,"url":"https://api.github.com/repos/explosion/spaCy","forks_url":"https://api.github.com/repos/explosion/spaCy/forks","keys_url":"https://api.github.com/repos/explosion/spaCy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/explosion/spaCy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/explosion/spaCy/teams","hooks_url":"https://api.github.com/repos/explosion/spaCy/hooks","issue_events_url":"https://api.github.com/repos/explosion/spaCy/issues/events{/number}","events_url":"https://api.github.com/repos/explosion/spaCy/events","assignees_url":"https://api.github.com/repos/explosion/spaCy/assignees{/user}","branches_url":"https://api.github.com/repos/explosion/spaCy/branches{/branch}","tags_url":"https://api.github.com/repos/explosion/spaCy/tags","blobs_url":"https://api.github.com/repos/explosion/spaCy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/explosion/spaCy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/explosion/spaCy/git/refs{/sha}","trees_url":"https://api.github.com/repos/explosion/spaCy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/explosion/spaCy/statuses/{sha}","languages_url":"https://api.github.com/repos/explosion/spaCy/languages","stargazers_url":"https://api.github.com/repos/explosion/spaCy/stargazers","contributors_url":"https://api.github.com/repos/explosion/spaCy/contributors","subscribers_url":"https://api.github.com/repos/explosion/spaCy/subscribers","subscription_url":"https://api.github.com/repos/explosion/spaCy/subscription","commits_url":"https://api.github.com/repos/explosion/spaCy/commits{/sha}","git_commits_url":"https://api.github.com/repos/explosion/spaCy/git/commits{/sha}","comments_url":"https://api.github.com/repos/explosion/spaCy/comments{/number}","issue_comment_url":"https://api.github.com/repos/explosion/spaCy/issues/comments{/number}","contents_url":"https://api.github.com/repos/explosion/spaCy/contents/{+path}","compare_url":"https://api.github.com/repos/explosion/spaCy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/explosion/spaCy/merges","archive_url":"https://api.github.com/repos/explosion/spaCy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/explosion/spaCy/downloads","issues_url":"https://api.github.com/repos/explosion/spaCy/issues{/number}","pulls_url":"https://api.github.com/repos/explosion/spaCy/pulls{/number}","milestones_url":"https://api.github.com/repos/explosion/spaCy/milestones{/number}","notifications_url":"https://api.github.com/repos/explosion/spaCy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/explosion/spaCy/labels{/name}","releases_url":"https://api.github.com/repos/explosion/spaCy/releases{/id}","deployments_url":"https://api.github.com/repos/explosion/spaCy/deployments","created_at":"2014-07-03T15:15:40Z","updated_at":"2017-10-18T07:21:01Z","pushed_at":"2017-10-18T03:26:43Z","git_url":"git://github.com/explosion/spaCy.git","ssh_url":"git@github.com:explosion/spaCy.git","clone_url":"https://github.com/explosion/spaCy.git","svn_url":"https://github.com/explosion/spaCy","homepage":"https://spacy.io","size":77381,"stargazers_count":5758,"watchers_count":5758,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":937,"mirror_url":null,"open_issues_count":180,"forks":937,"open_issues":180,"watchers":5758,"default_branch":"master"}},"base":{"label":"explosion:develop","ref":"develop","sha":"fc797a58de97cccfd5d99b2ecaf921c40ca33197","user":{"login":"explosion","id":20011530,"avatar_url":"https://avatars3.githubusercontent.com/u/20011530?v=4","gravatar_id":"","url":"https://api.github.com/users/explosion","html_url":"https://github.com/explosion","followers_url":"https://api.github.com/users/explosion/followers","following_url":"https://api.github.com/users/explosion/following{/other_user}","gists_url":"https://api.github.com/users/explosion/gists{/gist_id}","starred_url":"https://api.github.com/users/explosion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/explosion/subscriptions","organizations_url":"https://api.github.com/users/explosion/orgs","repos_url":"https://api.github.com/users/explosion/repos","events_url":"https://api.github.com/users/explosion/events{/privacy}","received_events_url":"https://api.github.com/users/explosion/received_events","type":"Organization","site_admin":false},"repo":{"id":21467110,"name":"spaCy","full_name":"explosion/spaCy","owner":{"login":"explosion","id":20011530,"avatar_url":"https://avatars3.githubusercontent.com/u/20011530?v=4","gravatar_id":"","url":"https://api.github.com/users/explosion","html_url":"https://github.com/explosion","followers_url":"https://api.github.com/users/explosion/followers","following_url":"https://api.github.com/users/explosion/following{/other_user}","gists_url":"https://api.github.com/users/explosion/gists{/gist_id}","starred_url":"https://api.github.com/users/explosion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/explosion/subscriptions","organizations_url":"https://api.github.com/users/explosion/orgs","repos_url":"https://api.github.com/users/explosion/repos","events_url":"https://api.github.com/users/explosion/events{/privacy}","received_events_url":"https://api.github.com/users/explosion/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/explosion/spaCy","description":"💫 Industrial-strength Natural Language Processing (NLP) with Python and Cython","fork":false,"url":"https://api.github.com/repos/explosion/spaCy","forks_url":"https://api.github.com/repos/explosion/spaCy/forks","keys_url":"https://api.github.com/repos/explosion/spaCy/keys{/key_id}","collaborators_url":"https://api.github.com/repos/explosion/spaCy/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/explosion/spaCy/teams","hooks_url":"https://api.github.com/repos/explosion/spaCy/hooks","issue_events_url":"https://api.github.com/repos/explosion/spaCy/issues/events{/number}","events_url":"https://api.github.com/repos/explosion/spaCy/events","assignees_url":"https://api.github.com/repos/explosion/spaCy/assignees{/user}","branches_url":"https://api.github.com/repos/explosion/spaCy/branches{/branch}","tags_url":"https://api.github.com/repos/explosion/spaCy/tags","blobs_url":"https://api.github.com/repos/explosion/spaCy/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/explosion/spaCy/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/explosion/spaCy/git/refs{/sha}","trees_url":"https://api.github.com/repos/explosion/spaCy/git/trees{/sha}","statuses_url":"https://api.github.com/repos/explosion/spaCy/statuses/{sha}","languages_url":"https://api.github.com/repos/explosion/spaCy/languages","stargazers_url":"https://api.github.com/repos/explosion/spaCy/stargazers","contributors_url":"https://api.github.com/repos/explosion/spaCy/contributors","subscribers_url":"https://api.github.com/repos/explosion/spaCy/subscribers","subscription_url":"https://api.github.com/repos/explosion/spaCy/subscription","commits_url":"https://api.github.com/repos/explosion/spaCy/commits{/sha}","git_commits_url":"https://api.github.com/repos/explosion/spaCy/git/commits{/sha}","comments_url":"https://api.github.com/repos/explosion/spaCy/comments{/number}","issue_comment_url":"https://api.github.com/repos/explosion/spaCy/issues/comments{/number}","contents_url":"https://api.github.com/repos/explosion/spaCy/contents/{+path}","compare_url":"https://api.github.com/repos/explosion/spaCy/compare/{base}...{head}","merges_url":"https://api.github.com/repos/explosion/spaCy/merges","archive_url":"https://api.github.com/repos/explosion/spaCy/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/explosion/spaCy/downloads","issues_url":"https://api.github.com/repos/explosion/spaCy/issues{/number}","pulls_url":"https://api.github.com/repos/explosion/spaCy/pulls{/number}","milestones_url":"https://api.github.com/repos/explosion/spaCy/milestones{/number}","notifications_url":"https://api.github.com/repos/explosion/spaCy/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/explosion/spaCy/labels{/name}","releases_url":"https://api.github.com/repos/explosion/spaCy/releases{/id}","deployments_url":"https://api.github.com/repos/explosion/spaCy/deployments","created_at":"2014-07-03T15:15:40Z","updated_at":"2017-10-18T07:21:01Z","pushed_at":"2017-10-18T03:26:43Z","git_url":"git://github.com/explosion/spaCy.git","ssh_url":"git@github.com:explosion/spaCy.git","clone_url":"https://github.com/explosion/spaCy.git","svn_url":"https://github.com/explosion/spaCy","homepage":"https://spacy.io","size":77381,"stargazers_count":5758,"watchers_count":5758,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":937,"mirror_url":null,"open_issues_count":180,"forks":937,"open_issues":180,"watchers":5758,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/explosion/spaCy/pulls/1425"},"html":{"href":"https://github.com/explosion/spaCy/pull/1425"},"issue":{"href":"https://api.github.com/repos/explosion/spaCy/issues/1425"},"comments":{"href":"https://api.github.com/repos/explosion/spaCy/issues/1425/comments"},"review_comments":{"href":"https://api.github.com/repos/explosion/spaCy/pulls/1425/comments"},"review_comment":{"href":"https://api.github.com/repos/explosion/spaCy/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/explosion/spaCy/pulls/1425/commits"},"statuses":{"href":"https://api.github.com/repos/explosion/spaCy/statuses/c0aceb9fbecfa0c62b3d3624b627a79e9984c040"}},"author_association":"OWNER"}}
{"actor":{"display_login":"ines"}}