{"action":"created","issue":{"url":"https://api.github.com/repos/thatdude624/Spark-Reader/issues/4","repository_url":"https://api.github.com/repos/thatdude624/Spark-Reader","labels_url":"https://api.github.com/repos/thatdude624/Spark-Reader/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/thatdude624/Spark-Reader/issues/4/comments","events_url":"https://api.github.com/repos/thatdude624/Spark-Reader/issues/4/events","html_url":"https://github.com/thatdude624/Spark-Reader/issues/4","id":215598249,"number":4,"title":"Question: kuromoji","user":{"login":"wareya","id":585488,"avatar_url":"https://avatars3.githubusercontent.com/u/585488?v=3","gravatar_id":"","url":"https://api.github.com/users/wareya","html_url":"https://github.com/wareya","followers_url":"https://api.github.com/users/wareya/followers","following_url":"https://api.github.com/users/wareya/following{/other_user}","gists_url":"https://api.github.com/users/wareya/gists{/gist_id}","starred_url":"https://api.github.com/users/wareya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wareya/subscriptions","organizations_url":"https://api.github.com/users/wareya/orgs","repos_url":"https://api.github.com/users/wareya/repos","events_url":"https://api.github.com/users/wareya/events{/privacy}","received_events_url":"https://api.github.com/users/wareya/received_events","type":"User","site_admin":false},"labels":[{"id":506566402,"url":"https://api.github.com/repos/thatdude624/Spark-Reader/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-21T00:51:49Z","updated_at":"2017-03-21T08:44:35Z","closed_at":null,"body":"I recently figured out how to use kuromoji, and theoretically speaking, I could see what happens if I force the segmenter to only create a segmentation if it's in a place where kuromoji put one. This would still allow the deconjugator to work, and it would fix cases like Nはこいつ, which my fork (and presumably the original) of spark reader currently splits wrong. Basically it would prevent the parser from making segments that are inconsistent with where kuromoji draws the boundaries between lexemes.\r\n\r\nIf I wanted to just fix this one example, I would probably add a way to say \"if はこ is followed by いつ, rewrite them into は followed by こいつ\", which is really awful. Either that or making the segmenter much smarter than it is right now, which would basically reimplement half of mecab or kuromoji.\r\n\r\nThe problem is, kuromoji is like, really big. The smallest version of it, kuromoji-ipadic, comes in at 13MB. If kuromoji were added in the way described above, it would probably be easy to make two builds of spark reader: one with kuromoji compiled in, one without it. If it were added in an invasive way that lets the segmenter rely on kuromoji more, you couldn't do that unless you were to encapsulate and abstract kuromoji's segmenting behavior and provide a \"dummy\" version of it if kuromoji is compiled out or not available.\r\n\r\nSpark reader's \"dumb\" segmenter is very good at separating terms that don't inflect and don't have confusing overlap, but kuromoji is good at syntactically complex stuff like strings of hiragana and conjugations. In the few cases kuromoji might do something wrong with separating terms, like complicated katakana names, I don't know the ways it might go wrong. Presumably, the fact that spark reader lets the user \"fix\" parsing mistakes would gloss over the problem.\r\n\r\nThere's also the fact that kuromoji is, itself, more code, so if it were added in a simple way, it would probably just slow down the segmenting process down. If kuromoji were added in an invasive way, it might actually make the segmenting process faster, since the only time you would have to look forward in kuromoji's segment list is if you're looking at a segment that can inflect (<-- big deal!)"},"comment":{"url":"https://api.github.com/repos/thatdude624/Spark-Reader/issues/comments/288011640","html_url":"https://github.com/thatdude624/Spark-Reader/issues/4#issuecomment-288011640","issue_url":"https://api.github.com/repos/thatdude624/Spark-Reader/issues/4","id":288011640,"user":{"login":"thatdude624","id":16175757,"avatar_url":"https://avatars1.githubusercontent.com/u/16175757?v=3","gravatar_id":"","url":"https://api.github.com/users/thatdude624","html_url":"https://github.com/thatdude624","followers_url":"https://api.github.com/users/thatdude624/followers","following_url":"https://api.github.com/users/thatdude624/following{/other_user}","gists_url":"https://api.github.com/users/thatdude624/gists{/gist_id}","starred_url":"https://api.github.com/users/thatdude624/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatdude624/subscriptions","organizations_url":"https://api.github.com/users/thatdude624/orgs","repos_url":"https://api.github.com/users/thatdude624/repos","events_url":"https://api.github.com/users/thatdude624/events{/privacy}","received_events_url":"https://api.github.com/users/thatdude624/received_events","type":"User","site_admin":false},"created_at":"2017-03-21T08:44:35Z","updated_at":"2017-03-21T08:44:35Z","body":"I'm alright with you adding alternative modes, but as you said perhaps make them all optional. I wouldn't know how to setup the build thing to only include some libraries sometimes and not cause compiler errors when those libraries are excluded. Right now, if you look at the code, JNA is always included and there's code in place that essentially disables it when on Linux (even though JNA isn't used for anything useful yet even on Windows). The new Epwing support works in a simmilar way.\r\n\r\nPerhaps don't initialise kuromoji if the user decides to use the internal splitter, or no splitter at all, but do still have it there.\r\n\r\nOn a related note, is it possible to add your new deconjugator alongside my 'legacy' one, and let the user choose which one? Yours would probably become the default, but I'd like to keep the old one even if only for testing purposes (unless that becomes hard to maintain due to other changes, then I'll have to drop it of course)"}}
{"actor":{"display_login":"thatdude624"}}