{"action":"created","issue":{"url":"https://api.github.com/repos/kpu/kenlm/issues/104","repository_url":"https://api.github.com/repos/kpu/kenlm","labels_url":"https://api.github.com/repos/kpu/kenlm/issues/104/labels{/name}","comments_url":"https://api.github.com/repos/kpu/kenlm/issues/104/comments","events_url":"https://api.github.com/repos/kpu/kenlm/issues/104/events","html_url":"https://github.com/kpu/kenlm/issues/104","id":259552864,"number":104,"title":"Can't train n-gram LM from large corpus","user":{"login":"mattkleonard","id":17750089,"avatar_url":"https://avatars2.githubusercontent.com/u/17750089?v=4","gravatar_id":"","url":"https://api.github.com/users/mattkleonard","html_url":"https://github.com/mattkleonard","followers_url":"https://api.github.com/users/mattkleonard/followers","following_url":"https://api.github.com/users/mattkleonard/following{/other_user}","gists_url":"https://api.github.com/users/mattkleonard/gists{/gist_id}","starred_url":"https://api.github.com/users/mattkleonard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattkleonard/subscriptions","organizations_url":"https://api.github.com/users/mattkleonard/orgs","repos_url":"https://api.github.com/users/mattkleonard/repos","events_url":"https://api.github.com/users/mattkleonard/events{/privacy}","received_events_url":"https://api.github.com/users/mattkleonard/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-09-21T16:14:49Z","updated_at":"2017-10-05T17:52:46Z","closed_at":null,"author_association":"NONE","body":"The input text file is 12GB, so large, but I don't think unmanageably massive.\r\nHere's the error:\r\n\r\n```\r\nbin/lmplz -o 5 -S 80% -T /tmp </Users/mattleonard/Documents/Python/language_models/wikipedia/wikitext_split.xml >/Users/mattleonard/Documents/Python/language_models/wikipedia/wikitext_split.arpa\r\n=== 1/5 Counting and sorting n-grams ===\r\nReading stdin\r\n----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100\r\n*****************Last input should have been poison.\r\nutil/file.cc:220 in void util::WriteOrThrow(int, const void *, std::size_t) threw FDException because `ret < 1'.\r\nInvalid argument in fd 4 while writing 5651839480 bytes\r\nAbort trap: 6\r\n```\r\n\r\nI should add that the same thing happens when training a trigram model."},"comment":{"url":"https://api.github.com/repos/kpu/kenlm/issues/comments/334542135","html_url":"https://github.com/kpu/kenlm/issues/104#issuecomment-334542135","issue_url":"https://api.github.com/repos/kpu/kenlm/issues/104","id":334542135,"user":{"login":"mattkleonard","id":17750089,"avatar_url":"https://avatars2.githubusercontent.com/u/17750089?v=4","gravatar_id":"","url":"https://api.github.com/users/mattkleonard","html_url":"https://github.com/mattkleonard","followers_url":"https://api.github.com/users/mattkleonard/followers","following_url":"https://api.github.com/users/mattkleonard/following{/other_user}","gists_url":"https://api.github.com/users/mattkleonard/gists{/gist_id}","starred_url":"https://api.github.com/users/mattkleonard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattkleonard/subscriptions","organizations_url":"https://api.github.com/users/mattkleonard/orgs","repos_url":"https://api.github.com/users/mattkleonard/repos","events_url":"https://api.github.com/users/mattkleonard/events{/privacy}","received_events_url":"https://api.github.com/users/mattkleonard/received_events","type":"User","site_admin":false},"created_at":"2017-10-05T17:52:46Z","updated_at":"2017-10-05T17:52:46Z","author_association":"NONE","body":"I'm re-upping this issue.\r\n\r\nSince originally posting, I've tried a few other things, and am still getting the error.\r\n\r\nIs it related to memory? I tried cutting my corpus in half and now it gets to ~55% (instead of ~25%) before throwing the \"poison\" error.\r\n\r\nThanks,\r\n\r\nMatt\r\n\r\n"}}
{"actor":{"display_login":"mattkleonard"}}