{"action":"created","issue":{"url":"https://api.github.com/repos/antlr/antlr4/issues/1613","repository_url":"https://api.github.com/repos/antlr/antlr4","labels_url":"https://api.github.com/repos/antlr/antlr4/issues/1613/labels{/name}","comments_url":"https://api.github.com/repos/antlr/antlr4/issues/1613/comments","events_url":"https://api.github.com/repos/antlr/antlr4/issues/1613/events","html_url":"https://github.com/antlr/antlr4/issues/1613","id":201590354,"number":1613,"title":"Extend MAX_DFA_EDGE to speed up lexer","user":{"login":"mdakin","id":1647551,"avatar_url":"https://avatars1.githubusercontent.com/u/1647551?v=4","gravatar_id":"","url":"https://api.github.com/users/mdakin","html_url":"https://github.com/mdakin","followers_url":"https://api.github.com/users/mdakin/followers","following_url":"https://api.github.com/users/mdakin/following{/other_user}","gists_url":"https://api.github.com/users/mdakin/gists{/gist_id}","starred_url":"https://api.github.com/users/mdakin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdakin/subscriptions","organizations_url":"https://api.github.com/users/mdakin/orgs","repos_url":"https://api.github.com/users/mdakin/repos","events_url":"https://api.github.com/users/mdakin/events{/privacy}","received_events_url":"https://api.github.com/users/mdakin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2017-01-18T14:35:23Z","updated_at":"2017-09-12T14:36:26Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nCurrently antlr lexer runtime uses a cache which is [limited to 127](https://github.com/antlr/antlr4/blob/9e2c0f76f913b48589da1319ee1edbd9a2cc1fe8/runtime/Java/src/org/antlr/v4/runtime/atn/LexerATNSimulator.java#L26)\r\n\r\nJava version:\r\n\r\n`public static final int MAX_DFA_EDGE = 127; // forces unicode to stay in ATN\r\n`\r\n\r\nThis default causes big slowdown if parsed document contains characters codes bigger than 127, which is common for a lot of latin scripts (from my tests, up to 3x  slowdown for Turkish text , >10x slowdown if most of the chars are out of range) I believe this should be a problem for Polish, German, Scandinavian texts as well.\r\n\r\nIf this range is extended to 240 (Latin-1) or even better to 368 ([Latin Extended A](https://en.wikipedia.org/wiki/Latin_script_in_Unicode)) This would extend fast lexing of mixed content documents to a lot more languages. \r\n\r\nI believe the price of increasing this cache size a little is worth the speed gains.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/antlr/antlr4/issues/comments/328872701","html_url":"https://github.com/antlr/antlr4/issues/1613#issuecomment-328872701","issue_url":"https://api.github.com/repos/antlr/antlr4/issues/1613","id":328872701,"user":{"login":"mdakin","id":1647551,"avatar_url":"https://avatars1.githubusercontent.com/u/1647551?v=4","gravatar_id":"","url":"https://api.github.com/users/mdakin","html_url":"https://github.com/mdakin","followers_url":"https://api.github.com/users/mdakin/followers","following_url":"https://api.github.com/users/mdakin/following{/other_user}","gists_url":"https://api.github.com/users/mdakin/gists{/gist_id}","starred_url":"https://api.github.com/users/mdakin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mdakin/subscriptions","organizations_url":"https://api.github.com/users/mdakin/orgs","repos_url":"https://api.github.com/users/mdakin/repos","events_url":"https://api.github.com/users/mdakin/events{/privacy}","received_events_url":"https://api.github.com/users/mdakin/received_events","type":"User","site_admin":false},"created_at":"2017-09-12T14:36:26Z","updated_at":"2017-09-12T14:36:26Z","author_association":"NONE","body":"@sharwell  I see, who decides on things like this? I can just throw a PR and see if it sticks, but  I don't want to waste my or anyone else's time."}}
{"actor":{"display_login":"mdakin"}}